Customising Your Models With TensorFlow 2

Lab 01 Layer Nodes
Lab 02 Device Placement
Lab 03 The Keras Functional API
Lab 04 Keras TimeseriesGenerator
Lab 05 Creating Datasets From Different Sources
Lab 06 TensorFlow Datasets
Lab 07 Data Pipeline
Lab 08 Tokenising Text Data
Lab 09 Stateful RNNs
Lab 10 Sequence Modelling
Lab 11 The Build Method
Lab 12 Tracking Metrics In Custom Training Loops
Lab 13 Model Subclassing And Custom Training Loops

Layer Nodes

In this reading, we will be looking at the concept of layer nodes when creating a computational graph with shared layers.

In [1]:

import tensorflow as tf
print(tf.__version__)

2.3.0

Creating a simple computational graph¶

You have previously seen how to construct multiple input or output models, and also how to access model layers. Let's start by creating two inputs:

In [2]:

# Create the input layers
from tensorflow.keras.layers import Input
a = Input(shape=(128, 128, 3), name = 'input_a')
b = Input(shape=(64, 64, 3), name = 'input_b')

Now, we create a 2D convolutional layer, and call it on one of the inputs.

In [3]:

# Create and use the convolutional layer
from tensorflow.keras.layers import Conv2D
conv = Conv2D(32, (6, 6), padding = 'same')
conv_out_a = conv(a)
print(type(conv_out_a))

<class 'tensorflow.python.framework.ops.Tensor'>

The output of the layer is now a new Tensor, which captures the operation of calling the layer conv on the input a.

By defining this new operation in our computational graph, we have added a node to the conv layer. This node relates the input tensor to the output tensor.

Layer input and outputs¶

We can retrieve the output of a layer using the output attribute, and we can also get the input by using the input attribute.

Similarly, we can retrieve the input/output shape using input_shape and output_shape.

In [4]:

# Print the input and output tensors
print(conv.input)
print(conv.output)

Tensor("input_a:0", shape=(None, 128, 128, 3), dtype=float32)
Tensor("conv2d/BiasAdd:0", shape=(None, 128, 128, 32), dtype=float32)

In [5]:

# Verify the input and output shapes
assert conv.input_shape == (None, 128, 128, 3)
assert conv.output_shape == (None, 128, 128, 32)

Creating a new layer node¶

Now, let's call this layer again on a different input:

In [6]:

# Call the layer a second time
conv_out_b = conv(b)

When we call the same layer multiple times, that layer owns multiple nodes indexed as 0, 1, 2...

Now, what happens if we call input and output for this layer?

In [7]:

# Check the input and output attributes
assert conv.input.name == a.name
assert conv.output.name == conv_out_a.name

As you can see, the layer's input is identified as being a and its output as being conved_a, something is going wrong here. As long as a layer is only connected to one input, there is no confusion about what should be the input, and .output will return the one output of the layer, but when the layer is called on multiple inputs we end up in an ambiguous situation.

Let's try to get the input/output shape:

In [8]:

# Try accessing the input_shape
print(conv.input_shape)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-48ff08249748> in <module>
      1 # Try accessing the input_shape
----> 2 print(conv.input_shape)

~\anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in input_shape(self)
   2131       return self._inbound_nodes[0].input_shapes
   2132     else:
-> 2133       raise AttributeError('The layer "' + str(self.name) +
   2134                            ' has multiple inbound nodes, '
   2135                            'with different input shapes. Hence '

AttributeError: The layer "conv2d has multiple inbound nodes, with different input shapes. Hence the notion of "input shape" is ill-defined for the layer. Use `get_input_shape_at(node_index)` instead.

In [9]:

# Try accessing the output_shape
print(conv.output_shape)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-9-417f475d989b> in <module>
      1 # Try accessing the output_shape
----> 2 print(conv.output_shape)

~\anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in output_shape(self)
   2184       return self._inbound_nodes[0].output_shapes
   2185     else:
-> 2186       raise AttributeError('The layer "%s"'
   2187                            ' has multiple inbound nodes, '
   2188                            'with different output shapes. Hence '

AttributeError: The layer "conv2d" has multiple inbound nodes, with different output shapes. Hence the notion of "output shape" is ill-defined for the layer. Use `get_output_shape_at(node_index)` instead.

input_shape and output_shape did not return the shape of the two inputs and outputs, instead they returned an error.

Indexing layer nodes¶

We have applied the same Conv2D layer to an input of shape (128, 128, 3), and then to an input of shape (64, 64, 3), therefore the layer has multiple input/output shapes, for this reason we now have to retrieve them by specifying the index of the node they belong to.

To get the inputs/outputs shapes, we now have to use get_input_shape_at and get_output_shape_at with the correct index:

In [10]:

# Print the input and output shapes for each layer node

assert conv.get_input_shape_at(0) == (None, 128, 128, 3)  # Tensor a
assert conv.get_input_shape_at(1) == (None, 64, 64, 3)  # Tensor b

assert conv.get_output_shape_at(0) == (None, 128, 128, 32)  # Tensor conv_out_a
assert conv.get_output_shape_at(1) == (None, 64, 64, 32)  # Tensor conv_out_b

Likewise, we use get_input_at and get_output_at to fetch the inputs/outputs:

In [11]:

assert conv.get_input_at(0).name == a.name
assert conv.get_input_at(1).name == b.name

assert conv.get_output_at(0).name == conv_out_a.name
assert conv.get_output_at(1).name == conv_out_b.name

Further reading and resources¶

• https://keras.io/getting-started/functional-api-guide/#the-concept-of-layer-node

Device Placement

In this reading, we are going to be looking at device placement. We will see how to access the device associated to a given tensor, and compare the use of GPUs and CPUs.

When running this notebook, ensure that the GPU runtime type is selected (Runtime -> Change runtime type).

In [1]:

import tensorflow as tf
tf.__version__

Out[1]:

'2.2.0'

Get the physical devices¶

First, we can list the physical devices available.

In [2]:

# List all physical devices
tf.config.list_physical_devices()

Out[2]:

[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'),
 PhysicalDevice(name='/physical_device:XLA_CPU:0', device_type='XLA_CPU'),
 PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'),
 PhysicalDevice(name='/physical_device:XLA_GPU:0', device_type='XLA_GPU')]

If you have enabled the GPU runtime, then you should see the GPU device in the above list.

We can also check specifically for the GPU or CPU devices.

In [3]:

# Check for GPU devices
tf.config.list_physical_devices('GPU')

Out[3]:

[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

In [4]:

# Check for CPU devices
tf.config.list_physical_devices('CPU')

Out[4]:

[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]

We can get the GPU device name as follows:

In [5]:

# Get the GPU device name
tf.test.gpu_device_name()

Out[5]:

'/device:GPU:0'

Placement of Tensor operations¶

TensorFlow will automatically allocate Tensor operations to a physical device, and will handle the copying between CPU and GPU memory if necessary.

Let's define a random Tensor:

In [6]:

# Define a Tensor
x = tf.random.uniform([3, 3])

We can see which device this Tensor is placed on using its device attribute.

In [7]:

# Get the Tensor device
x.device

Out[7]:

'/job:localhost/replica:0/task:0/device:GPU:0'

The above string will end with 'GPU:K' if the Tensor is placed on the K-th GPU device. We can also check if a tensor is placed on a specific device by using device_endswith:

In [8]:

# Test for device allocation
print("Is the Tensor on CPU #0:  "),
print(x.device.endswith('CPU:0'))
print('')
print("Is the Tensor on GPU #0:  "),
print(x.device.endswith('GPU:0'))

Is the Tensor on CPU #0:  
False

Is the Tensor on GPU #0:  
True

Specifying device placement¶

As mentioned previously, TensorFlow will automatically allocate Tensor operations to specific devices. However, it is possible to force placement on specific devices, if they are available.

We can view the benefits of GPU acceleration by running some tests and placing the operations on the CPU or GPU respectively.

In [9]:

# Define simple tests to time computation speed
import time

def time_matadd(x):
    start = time.time()
    for loop in range(10):
        tf.add(x, x)
    result = time.time()-start
    print("Matrix addition (10 loops): {:0.2f}ms".format(1000*result))


def time_matmul(x):
    start = time.time()
    for loop in range(10):
        tf.matmul(x, x)
    result = time.time()-start
    print("Matrix multiplication (10 loops): {:0.2f}ms".format(1000*result))

In the following cell, we run the above tests inside the context with tf.device("CPU:0"), which forces the operations to be run on the CPU.

In [10]:

# Force execution on CPU
print("On CPU:")
with tf.device("CPU:0"):
    x = tf.random.uniform([1000, 1000])
    assert x.device.endswith("CPU:0")
    time_matadd(x)
    time_matmul(x)

On CPU:
Matrix addition (10 loops): 12.45ms
Matrix multiplication (10 loops): 125.25ms

And now run the same operations on the GPU:

In [11]:

# Force execution on GPU #0 if available
if tf.config.experimental.list_physical_devices("GPU"):
    print("On GPU:")
    with tf.device("GPU:0"): 
        x = tf.random.uniform([1000, 1000])
        assert x.device.endswith("GPU:0")
        time_matadd(x)
        time_matmul(x)

On GPU:
Matrix addition (10 loops): 0.74ms
Matrix multiplication (10 loops): 383.67ms

Note the significant time difference between running these operations on different devices.

Model training¶

Finally, we will demonstrate that GPU device placement offers speedup benefits for model training.

In [12]:

# Load the MNIST dataset
from tensorflow.keras.datasets import mnist
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train/255., x_test/255.

In [13]:

# Reduce the dataset size to speed up the test
x_train, y_train = x_train[:1000], y_train[:1000]

In [14]:

# Define a function to build the model
from tensorflow.keras import layers
from tensorflow.keras.models import  Sequential

def get_model():
  model = Sequential([
      layers.Conv2D(32, (3, 3), activation='relu', padding='same', input_shape=(28, 28, 1)),
      layers.MaxPooling2D((2, 2)),
      layers.Conv2D(64, (3, 3), activation='relu', padding='same'),
      layers.MaxPooling2D((2, 2)),
      layers.Conv2D(128, (3, 3), activation='relu', padding='same'),
      layers.MaxPooling2D((2, 2)),
      layers.Flatten(),
      layers.Dense(64, activation='relu'),
      layers.Dense(10, activation='softmax'),
      ])
  return model

In [15]:

# Time a training run on the CPU
with tf.device("CPU:0"):
  model = get_model()
  model.compile (
      optimizer = tf.keras.optimizers.RMSprop(1e-3),
      loss = 'sparse_categorical_crossentropy',
      metrics=['accuracy']
  )
  start = time.time()
  model.fit(x_train[..., np.newaxis], y_train, epochs=5, verbose=0)
  result = time.time() - start

print("CPU training time: {:0.2f}ms".format(1000 * result))

CPU training time: 3496.54ms

In [16]:

# Time a training run on the GPU
with tf.device("GPU:0"):
  model = get_model()
  model.compile (
      optimizer = tf.keras.optimizers.RMSprop(1e-3),
      loss = 'sparse_categorical_crossentropy',
      metrics = ['accuracy']
  )
  start = time.time()
  model.fit(x_train[..., np.newaxis], y_train, epochs=5, verbose=0)
  result = time.time() - start

print("GPU training time: {:0.2f}ms".format(1000 * result))

GPU training time: 3114.81ms

Further reading and resources¶

• https://www.tensorflow.org/tutorials/customization/basics#gpu_acceleration

The Keras Functional API

In [1]:

import tensorflow as tf
tf.__version__

Out[1]:

'2.2.0'

1. Multiple inputs and outputs¶

2. Tensors and Variables¶

3. Accessing model layers¶

4. Freezing layers¶

---

Multiple inputs and outputs¶

In [2]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

Load the acute inflammations dataset¶

The acute inflammations was created by a medical expert as a data set to test the expert system, which will perform the presumptive diagnosis of two diseases of the urinary system. You can find out more about the dataset here.

Attribute information:

Inputs:

• Temperature of patient : 35C-42C
• Occurrence of nausea : yes/no
• Lumbar pain : yes/no
• Urine pushing (continuous need for urination) : yes/no
• Micturition pains : yes/no
• Burning of urethra, itch, swelling of urethra outlet : yes/no

Outputs:

• decision 1: Inflammation of urinary bladder : yes/no
• decision 2: Nephritis of renal pelvis origin : yes/no

Load the data¶

In [3]:

# Load the dataset
from sklearn.model_selection import train_test_split
pd_dat = pd.read_csv('data/diagnosis.csv')
dataset = pd_dat.values

In [4]:

# Build train and test data splits
X_train, X_test, Y_train, Y_test = train_test_split(dataset[:,:6], dataset[:,6:], test_size=0.33)

In [5]:

# Assign training and testing inputs/outputs
temp_train, nocc_train, lumbp_train, up_train, mict_train, bis_train = np.transpose(X_train)
temp_test, nocc_test, lumbp_test, up_test, mict_test, bis_test = np.transpose(X_test)
inflam_train, nephr_train = Y_train[:, 0], Y_train[:, 1]
inflam_test, nephr_test = Y_test[:, 0], Y_test[:, 1]

Build the model¶

In [6]:

# Build the input layers
from tensorflow.keras import Input, layers
shape_inputs = (1,)
temperature = Input(shape = shape_inputs, name = 'temp')
nausea_occurence = Input(shape = shape_inputs, name = 'nocc')
lumbar_pain = Input(shape = shape_inputs, name = 'lumbp')
urine_pushing = Input(shape = shape_inputs, name = 'up')
micturition_pains = Input(shape = shape_inputs, name = 'mict')
bis = Input(shape = shape_inputs, name = 'bis')

In [7]:

# Create a list of all the inputs
list_inputs = [temperature, nausea_occurence, lumbar_pain, urine_pushing, 
               micturition_pains, bis]

In [8]:

# Merge all input features into a single large vector
x = layers.concatenate(list_inputs)

In [9]:

# Use a logistic regression classifier for disease prediction
inflammation_pred = layers.Dense(1, activation = 'sigmoid', name = 'inflam')(x)
nephritis_pred = layers.Dense(1, activation = 'sigmoid', name = 'nephr')(x)

In [10]:

# Create a list of all the outputs
list_outputs = [inflammation_pred, nephritis_pred]

In [11]:

# Create the model object
model = tf.keras.Model(inputs = list_inputs, outputs = list_outputs)

Plot the model¶

In [12]:

# Display the multiple input/output model
tf.keras.utils.plot_model(model, 'multi_input_output_model.png', show_shapes = True)

Out[12]:

Compile the model¶

In [13]:

# Compile the model
model.compile (
    optimizer = tf.keras.optimizers.RMSprop(1e-3),
    loss = ['binary_crossentropy', 'binary_crossentropy'],
    metrics = {'inflam': ['acc'], 'nephr': ['acc']},
    loss_weights = [1., 0.2]
)

Fit the model¶

In [14]:

# Define training inputs and outputs
inputs_train = {
    'temp': temp_train, 'nocc': nocc_train, 'lumbp': lumbp_train,
    'up': up_train, 'mict': mict_train, 'bis': bis_train
}
outputs_train = {'inflam': inflam_train, 'nephr': nephr_train}

In [15]:

# Train the model
history = model.fit(inputs_train, outputs_train, epochs = 1000, batch_size = 128, verbose = 0)

Plot the learning curves¶

In [16]:

# Plot the training accuracy

acc_keys = [k for k in history.history.keys() if k in ('inflam_acc', 'nephr_acc')] 
loss_keys = [k for k in history.history.keys() if not k in acc_keys]

for k, v in history.history.items():
    if k in acc_keys:
        plt.figure(1)
        plt.plot(v)
    else:
        plt.figure(2)
        plt.plot(v)

plt.figure(1)
plt.title('Accuracy vs. epochs')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(acc_keys, loc='upper right')

plt.figure(2)
plt.title('Loss vs. epochs')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(loss_keys, loc='upper right')

plt.show()

In [17]:

# Evaluate the model
model.evaluate (
    [temp_test, nocc_test, lumbp_test, up_test, mict_test, bis_test],
    [inflam_test, nephr_test],
    verbose = 2
)

2/2 - 0s - loss: 0.4909 - inflam_loss: 0.3909 - nephr_loss: 0.5000 - inflam_acc: 1.0000 - nephr_acc: 0.9250

Out[17]:

[0.4909321367740631,
 0.39093565940856934,
 0.49998220801353455,
 1.0,
 0.925000011920929]

---

Tensors and Variables¶

In [18]:

import numpy as np

Create Variable objects¶

In [19]:

# Create Variable objects of different type with tf.Variable

strings = tf.Variable(['Hello world!'], tf.string)
floats  = tf.Variable([3.14159, 2.71828], tf.float64)
ints = tf.Variable([1, 2, 3], tf.int32)
complexs = tf.Variable([25.9 - 7.39j, 1.23 - 4.91j], tf.complex128)

In [20]:

# Initialise a Variable value
tf.Variable(tf.constant(4.2, shape = (3, 3)))

Out[20]:

<tf.Variable 'Variable:0' shape=(3, 3) dtype=float32, numpy=
array([[4.2, 4.2, 4.2],
       [4.2, 4.2, 4.2],
       [4.2, 4.2, 4.2]], dtype=float32)>

Use and modify Variable values¶

In [21]:

# Use the value of a Variable
v = tf.Variable(0.0)
w = v + 1  # w is a tf.Tensor which is computed based on the value of v.
print(type(w))

<class 'tensorflow.python.framework.ops.EagerTensor'>

In [22]:

# Increment the value of a Variable
v.assign_add(1)
v

Out[22]:

<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=1.0>

In [23]:

# Decrement the value of a Variable
v.assign_sub(1)
v

Out[23]:

<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=0.0>

Create Tensor objects¶

Create a constant tensor and print its type as well as its shape:

In [24]:

# Create a constant Tensor
x = tf.constant([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(x)
print('dtype:', x.dtype)
print('shape:', x.shape)

tf.Tensor(
[[1 2 3]
 [4 5 6]
 [7 8 9]], shape=(3, 3), dtype=int32)
dtype: <dtype: 'int32'>
shape: (3, 3)

In [25]:

# Obtain the value as a numpy array
x.numpy()

Out[25]:

array([[1, 2, 3],
       [4, 5, 6],
       [7, 8, 9]], dtype=int32)

In [26]:

# Create a Tensor of type float32
x = tf.constant([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype = tf.float32)
x.dtype

Out[26]:

tf.float32

In [27]:

# Create coefficients
coeffs = np.arange(16)

In [28]:

# Initialise shapes
shape1 = [8, 2]
shape2 = [4, 4]
shape3 = [2, 2, 2, 2]

In [29]:

# Create Tensors of different shape
a = tf.constant(coeffs, shape=shape1)
print("\n a:\n ", a)
b = tf.constant(coeffs, shape=shape2)
print("\n b:\n ", b)
c = tf.constant(coeffs, shape=shape3)
print("\n c:\n ", c)

 a:
  tf.Tensor(
[[ 0  1]
 [ 2  3]
 [ 4  5]
 [ 6  7]
 [ 8  9]
 [10 11]
 [12 13]
 [14 15]], shape=(8, 2), dtype=int64)

 b:
  tf.Tensor(
[[ 0  1  2  3]
 [ 4  5  6  7]
 [ 8  9 10 11]
 [12 13 14 15]], shape=(4, 4), dtype=int64)

 c:
  tf.Tensor(
[[[[ 0  1]
   [ 2  3]]

  [[ 4  5]
   [ 6  7]]]


 [[[ 8  9]
   [10 11]]

  [[12 13]
   [14 15]]]], shape=(2, 2, 2, 2), dtype=int64)

Useful Tensor operations¶

In [30]:

# Create a constant Tensor
t = tf.constant(np.arange(80), shape=[5, 2, 8])

In [31]:

# Get the rank of a Tensor
rank = tf.rank(t)

In [32]:

# Display the rank
print("rank: ", rank)

rank:  tf.Tensor(3, shape=(), dtype=int32)

In [33]:

# Reshape a Tensor
t2 = tf.reshape(t, [8, 10])

In [34]:

# Display the new shape
print("t2.shape: ", t2.shape)

t2.shape:  (8, 10)

In [35]:

# Create ones, zeros, identity and constant Tensors
ones = tf.ones(shape = (2, 3))
zeros = tf.zeros(shape = (2, 4))
eye = tf.eye(3)
tensor7 = tf.constant(7.0, shape = [2, 2])

In [36]:

# Display the created tensors

print("\n Ones:\n ", ones)
print("\n Zeros:\n ", zeros)
print("\n Identity:\n ", eye)
print("\n Tensor filled with 7: ", tensor7)

 Ones:
  tf.Tensor(
[[1. 1. 1.]
 [1. 1. 1.]], shape=(2, 3), dtype=float32)

 Zeros:
  tf.Tensor(
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]], shape=(2, 4), dtype=float32)

 Identity:
  tf.Tensor(
[[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]], shape=(3, 3), dtype=float32)

 Tensor filled with 7:  tf.Tensor(
[[7. 7.]
 [7. 7.]], shape=(2, 2), dtype=float32)

In [37]:

# Create a ones Tensor and a zeros Tensor
t1 = tf.ones(shape = (2, 2))
t2 = tf.zeros(shape = (2, 2))

In [38]:

# Concatentate two Tensors
concat0 = tf.concat([t1, t2], 0)
concat1 = tf.concat([t1, t2], 1)

In [39]:

# Display the concatenated tensors
print(concat0)
print(concat1)

tf.Tensor(
[[1. 1.]
 [1. 1.]
 [0. 0.]
 [0. 0.]], shape=(4, 2), dtype=float32)
tf.Tensor(
[[1. 1. 0. 0.]
 [1. 1. 0. 0.]], shape=(2, 4), dtype=float32)

In [40]:

# Create a constant Tensor
t = tf.constant(np.arange(24), shape = (3, 2, 4))
print("\n t shape: ", t.shape)

 t shape:  (3, 2, 4)

In [41]:

# Expanding the rank of Tensors
t1 = tf.expand_dims(t, 0)
t2 = tf.expand_dims(t, 1)
t3 = tf.expand_dims(t, 3)

In [42]:

# Display the shapes after tf.expand_dims
print (
    "\n After expanding dims:\n t1 shape: ", t1.shape,
    "\n t2 shape: ", t2.shape,
    "\n t3 shape: ", t3.shape
)

 After expanding dims:
 t1 shape:  (1, 3, 2, 4) 
 t2 shape:  (3, 1, 2, 4) 
 t3 shape:  (3, 2, 4, 1)

In [43]:

# Squeezing redundant dimensions
t1 = tf.squeeze(t1, 0)
t2 = tf.squeeze(t2, 1)
t3 = tf.squeeze(t3, 3)

In [44]:

# Display the shapes after tf.squeeze
print (
    "\n After squeezing:\n t1 shape: ", t1.shape,
    "\n t2 shape: ", t2.shape,
    "\n t3 shape: ", t3.shape
)

 After squeezing:
 t1 shape:  (3, 2, 4) 
 t2 shape:  (3, 2, 4) 
 t3 shape:  (3, 2, 4)

In [45]:

# Slicing a Tensor
x = tf.constant([1, 2, 3, 4, 5, 6, 7])
print(x[1:4])

tf.Tensor([2 3 4], shape=(3,), dtype=int32)

Doing maths with Tensors¶

In [46]:

# Create two constant Tensors
c = tf.constant([[1.0, 2.0], [3.0, 4.0]])
d = tf.constant([[1.0, 1.0], [0.0, 1.0]])

In [47]:

# Matrix multiplication
matmul_cd = tf.matmul(c, d)

In [48]:

# Display the result
print("\n tf.matmul(c,d):\n", matmul_cd)

 tf.matmul(c,d):
 tf.Tensor(
[[1. 3.]
 [3. 7.]], shape=(2, 2), dtype=float32)

In [49]:

# Elementwise operations
c_times_d = c * d
c_plus_d = c + d
c_minus_d = c - d
c_div_c = c / c

In [50]:

# Display the results
print("\n c*d:\n", c_times_d)
print("\n c+d:\n", c_plus_d)
print("\n c-d:\n", c_minus_d)
print("\n c/c:\n", c_div_c)

 c*d:
 tf.Tensor(
[[1. 2.]
 [0. 4.]], shape=(2, 2), dtype=float32)

 c+d:
 tf.Tensor(
[[2. 3.]
 [3. 5.]], shape=(2, 2), dtype=float32)

 c-d:
 tf.Tensor(
[[0. 1.]
 [3. 3.]], shape=(2, 2), dtype=float32)

 c/c:
 tf.Tensor(
[[1. 1.]
 [1. 1.]], shape=(2, 2), dtype=float32)

In [51]:

# Create Tensors
a = tf.constant([[2, 3], [3, 3]])
b = tf.constant([[8, 7], [2, 3]])
x = tf.constant([[-6.89 + 1.78j], [-2.54 + 2.15j]])

In [52]:

# Absolute value of a Tensor
absx = tf.abs(x)

In [53]:

# Power of a Tensor
powaa = tf.pow(a, a)

In [54]:

# Display the results
print("\n ", absx)
print("\n ", powaa)

  tf.Tensor(
[[7.11621388]
 [3.32777704]], shape=(2, 1), dtype=float64)

  tf.Tensor(
[[ 4 27]
 [27 27]], shape=(2, 2), dtype=int32)

Randomly sampled constant tensors¶

In [55]:

# Create a Tensor with samples from a Normal distribution
tn = tf.random.normal(shape = (2, 2), mean = 0, stddev = 1.)
tn

Out[55]:

<tf.Tensor: shape=(2, 2), dtype=float32, numpy=
array([[ 1.1122508 ,  0.02933991],
       [ 0.45304555, -2.482614  ]], dtype=float32)>

In [56]:

# Create a Tensor with samples from a Uniform distribution
tu = tf.random.uniform(shape = (2, 1), minval = 0, maxval = 10, dtype = 'int32')
tu

Out[56]:

<tf.Tensor: shape=(2, 1), dtype=int32, numpy=
array([[9],
       [7]], dtype=int32)>

In [57]:

# Create a Tensor with samples from a Poisson distribution
tu = tf.random.poisson((2, 2), 5)
tu

Out[57]:

<tf.Tensor: shape=(2, 2), dtype=float32, numpy=
array([[4., 4.],
       [4., 6.]], dtype=float32)>

In [58]:

# More maths operations
d = tf.square(tn)
e = tf.exp(d)
f = tf.cos(c)
print(d)
print(e)
print(f)

tf.Tensor(
[[1.2371019e+00 8.6083042e-04]
 [2.0525026e-01 6.1633725e+00]], shape=(2, 2), dtype=float32)
tf.Tensor(
[[  3.4456134   1.0008612]
 [  1.2278323 475.0274   ]], shape=(2, 2), dtype=float32)
tf.Tensor(
[[ 0.5403023 -0.4161468]
 [-0.9899925 -0.6536436]], shape=(2, 2), dtype=float32)

---

Accessing model layers¶

In [59]:

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

Load the pre-trained model¶

In this section, we aim to demonstrate accessing layer attributes within a model.

Let's get started by loading the VGG19 pre-trained model from the keras.applications library, which is a very deep network trained on more than a million images from the ImageNet database. The network is trained to classify images into 1000 object categories.

In [60]:

# Load the VGG19 model

from tensorflow.keras.applications import VGG19
# vgg_model = VGG19()
vgg_model = VGG19(weights = None)
vgg_model.load_weights('models/vgg19_weights_tf_dim_ordering_tf_kernels.h5')

In [61]:

# Get the inputs, layers and display the summary
vgg_input = vgg_model.input
vgg_layers = vgg_model.layers
vgg_model.summary()

Model: "vgg19"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_conv4 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_conv4 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv4 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0         
_________________________________________________________________
fc1 (Dense)                  (None, 4096)              102764544 
_________________________________________________________________
fc2 (Dense)                  (None, 4096)              16781312  
_________________________________________________________________
predictions (Dense)          (None, 1000)              4097000   
=================================================================
Total params: 143,667,240
Trainable params: 143,667,240
Non-trainable params: 0
_________________________________________________________________

Build a model to access the layer outputs¶

In [62]:

from tensorflow.keras.models import Model

In [63]:

# Build a model that returns the layer outputs
layer_outputs = [layer.output for layer in vgg_layers]
features = Model(inputs = vgg_input, outputs = layer_outputs)

In [64]:

# Plot the model
tf.keras.utils.plot_model(features, 'vgg19_model.png', show_shapes = True)

Out[64]:

In [65]:

# Test the model on a random input
img = np.random.random((1, 224, 224, 3)).astype('float32')
extracted_features = features(img)

Load the 'cool cat' picture¶

In Zambia’s South Luangwa National Park, a photographer had been watching a pride of lions while they slept off a feast from a buffalo kill. When this female walked away, he anticipated that she might be going for a drink and so he positioned his vehicle on the opposite side of the waterhole. The cool cat picture is one of the highly commended 2018 Image from Wildlife Photographer of the Year.

In [66]:

# Display the original image
import IPython.display as display
from PIL import Image
display.display(Image.open('data/cool_cat.jpg'))

Visualise network features from the input image¶

In [67]:

# Preprocess the image

from tensorflow.keras.applications.vgg19 import preprocess_input
from tensorflow.keras.preprocessing import image

img_path = 'data/cool_cat.jpg'
img = image.load_img(img_path, target_size = (224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis = 0)
x = preprocess_input(x)

In [68]:

# Extract the features
extracted_features = features(x)

In [69]:

# Visualise the input channels
f1 = extracted_features[0]
print('\n f1.shape: ', f1.shape)

imgs = f1[0, :, :]
plt.figure(figsize = (15, 15))
for n in range(3):
    ax = plt.subplot(1, 3, n + 1)
    plt.imshow(imgs[:, :, n])
    plt.axis('off')
plt.subplots_adjust(wspace = 0.01, hspace = 0.01)

 f1.shape:  (1, 224, 224, 3)

In [70]:

# Visualise some features in the first hidden layer
f2 = extracted_features[1]
print('\n f2.shape: ', f2.shape)

imgs = f2[0, :, :]
plt.figure(figsize = (15, 15))
for n in range(16):
    ax = plt.subplot(4, 4, n + 1)
    plt.imshow(imgs[:, :, n])
    plt.axis('off')
plt.subplots_adjust(wspace = 0.01, hspace = 0.01)

 f2.shape:  (1, 224, 224, 64)

In [71]:

# Build a model to extract features by layer name
extracted_features_block1_pool = \
    Model(inputs = features.input, outputs = features.get_layer('block1_pool').output)
block1_pool_features = extracted_features_block1_pool.predict(x)

In [72]:

# Visualise some features from the extracted layer output
imgs = block1_pool_features[0, :, :]
plt.figure(figsize = (15, 15))
for n in range(16):
    ax = plt.subplot(4, 4, n + 1)
    plt.imshow(imgs[:, :, n])
    plt.axis('off')
plt.subplots_adjust(wspace = 0.01, hspace = 0.01)

In [73]:

# Extract features from a layer deeper in the network
extracted_features_block5_conv4 = \
    Model(inputs = features.input, outputs = features.get_layer('block5_conv4').output)
block5_conv4_features = extracted_features_block5_conv4.predict(x)

In [74]:

# Visualise some features from the extracted layer output
imgs = block5_conv4_features[0, :, :]
plt.figure(figsize = (15, 15))
for n in range(16):
    ax = plt.subplot(4, 4, n + 1)
    plt.imshow(imgs[:, :, n])
    plt.axis('off')
plt.subplots_adjust(wspace = 0.01, hspace = 0.01)

---

Freezing layers¶

In [75]:

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

Build the model¶

In [76]:

# Build a small Sequential model
from tensorflow.keras.models import Sequential
from tensorflow.keras import layers
model = Sequential ([
    layers.Dense (
        4, input_shape = (4,), activation = 'relu',
        kernel_initializer = 'random_uniform', bias_initializer = 'ones'
    ),
    layers.Dense(2, activation = 'relu', kernel_initializer = 'lecun_normal', bias_initializer = 'ones'),
    layers.Dense(4, activation = 'softmax'),
])

In [77]:

# Display the model summary
model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 4)                 20        
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 10        
_________________________________________________________________
dense_2 (Dense)              (None, 4)                 12        
=================================================================
Total params: 42
Trainable params: 42
Non-trainable params: 0
_________________________________________________________________

Examine the weight matrix variation over training¶

In [78]:

def get_weights(model):
    return [e.weights[0].numpy() for e in model.layers]

def get_biases(model):
    return [e.bias.numpy() for e in model.layers]

def plot_delta_weights(W0_layers, W1_layers, b0_layers, b1_layers):
    plt.figure(figsize = (8, 8))
    for n in range(3):
        delta_l = W1_layers[n] - W0_layers[n]
        print('Layer ' + str(n) + ': bias variation: ', np.linalg.norm(b1_layers[n] - b0_layers[n]))
        ax = plt.subplot(1, 3, n + 1)
        plt.imshow(delta_l)
        plt.title('Layer ' + str(n))
        plt.axis('off')
    plt.colorbar()
    plt.suptitle('Weight matrices variation')

In [79]:

# Retrieve the weights and biases
W0_layers = get_weights(model)
b0_layers = get_biases(model)

In [80]:

# Construct a synthetic dataset
x_train = np.random.random((100, 4))
y_train = x_train
x_test = np.random.random((20, 4))
y_test = x_test

In [81]:

# Compile and fit the model
model.compile(optimizer = 'adam', loss = 'mse', metrics = ['acc'])
model.fit(x_train, y_train, epochs = 50, verbose = False);

In [82]:

# Retrieve weights and biases
W1_layers = get_weights(model)
b1_layers = get_biases(model)

In [83]:

# Plot the variation
plot_delta_weights(W0_layers, W1_layers, b0_layers, b1_layers)

Layer 0: bias variation:  0.028452259
Layer 1: bias variation:  0.014293373
Layer 2: bias variation:  0.092317775

Freeze layers at build time¶

In [84]:

# Count the trainable and non trainable variables before the freezing
n_trainable_variables = len(model.trainable_variables)
n_non_trainable_variables = len(model.non_trainable_variables)

In [85]:

# Display the number of trainable and non trainable variables before the freezing
print (
    "\n Before freezing:\n\t Number of trainable variables: ", n_trainable_variables,
    "\n\t Number of non trainable variables: ", n_non_trainable_variables
)

 Before freezing:
	 Number of trainable variables:  6 
	 Number of non trainable variables:  0

In [86]:

# Build the model
model = Sequential ([
    layers.Dense (
        4, input_shape = (4,), activation = 'relu',
        kernel_initializer = 'random_uniform', bias_initializer = 'ones', trainable = False
    ),
    layers.Dense(2, activation = 'relu', kernel_initializer = 'lecun_normal', bias_initializer = 'ones'),
    layers.Dense(4, activation = 'softmax'),
])

In [87]:

# Count the trainable and non trainable variables after the freezing
n_trainable_variables = len(model.trainable_variables)
n_non_trainable_variables = len(model.non_trainable_variables)

In [88]:

# Display the number of trainable and non trainable variables after the freezing
print (
    "\n After freezing:\n\t Number of trainable variables: ", n_trainable_variables,
    "\n\t Number of non trainable variables: ", n_non_trainable_variables
)

 After freezing:
	 Number of trainable variables:  4 
	 Number of non trainable variables:  2

In [89]:

# Retrieve weights and biases
W0_layers = get_weights(model)
b0_layers = get_biases(model)

In [90]:

# Compile and fit the model
model.compile(optimizer = 'adam', loss = 'mse', metrics = ['acc'])
model.fit(x_train, y_train, epochs = 50, verbose = False);

In [91]:

# Retrieve weights and biases
W1_layers = get_weights(model)
b1_layers = get_biases(model)

In [92]:

# Plot the variation
plot_delta_weights(W0_layers, W1_layers, b0_layers, b1_layers)

Layer 0: bias variation:  0.0
Layer 1: bias variation:  0.1509809
Layer 2: bias variation:  0.33442858

Freeze layers of a pre-built model¶

In [93]:

# Count the trainable and non trainable variables before the freezing
print (
    "\n Before freezing:\n\t Number of trainable variables: ", len(model.trainable_variables),
    "\n\t Number of non trainable variables: ", len(model.non_trainable_variables)
)

 Before freezing:
	 Number of trainable variables:  4 
	 Number of non trainable variables:  2

In [94]:

# Freeze the second layer
model.layers[1].trainable = False

In [95]:

# Count the trainable and non trainable variables after the freezing
print (
    "\n After freezing:\n\t Number of trainable variables: ", len(model.trainable_variables),
    "\n\t Number of non trainable variables: ", len(model.non_trainable_variables)
)

 After freezing:
	 Number of trainable variables:  2 
	 Number of non trainable variables:  4

In [96]:

# Compile and fit the model
model.compile(optimizer = 'adam', loss = 'mse', metrics = ['acc'])
model.fit(x_train, y_train, epochs = 50, verbose = False);

In [97]:

# Retrieve weights and biases
W2_layers = get_weights(model)
b2_layers = get_biases(model)

In [98]:

# Plot the variation
plot_delta_weights(W1_layers, W2_layers, b1_layers, b2_layers)

Layer 0: bias variation:  0.0
Layer 1: bias variation:  0.0
Layer 2: bias variation:  0.27730897

Keras TimeseriesGenerator

In this reading we'll be looking at the TimeseriesGenerator, which is used for preprocessing and generating batches of temporal data.

Examples of sequential data are audio tracks, music, books and essays. Here, the order of the notes, words and sentences carry information about the meaning.

In [1]:

import tensorflow as tf
tf.__version__

Out[1]:

'2.2.0'

The dataset¶

In this notebook, we'll be using the DSD100 dataset to demonstrate the use of the TimeseriesGenerator to perform various preprocessing operations. The DSD100 dataset is a dataset of 100 different music tracks in different styles. Its intended use is signal separation, and it also includes the separate instrument tracks that add up to make the music tracks. The tracks are all stereophonic and sampled at 44.1kHz. A sample from the dataset is provided.

Run the cell below to load one of the sample songs and press the play button to listen to the song.

In [2]:

# Play a sample track from the DSD100 dataset

from IPython import display as ipd

ipd.display(ipd.Audio("data/055 - Angels In Amplifiers - I'm Alright/mixture.wav"))

TimeseriesGenerator¶

Before diving into working with preprocessing techniques on the DSD100 dataset, let's start by getting grips with the operations that we can perform using the TimeseriesGenerator on simple synthetic data.

We'll begin by defining a simple time series dataset and corresponding sequence of targets.

In [3]:

# Create a simple time series dataset

import numpy as np

dummy_data = np.arange(1, 11, 1)
dummy_targets = np.arange(10, 110, 10)
print(dummy_data)
print(dummy_targets)

[ 1  2  3  4  5  6  7  8  9 10]
[ 10  20  30  40  50  60  70  80  90 100]

The TimeseriesGenerator has three required arguments: data, targets and length.

The data argument could be a list of numpy array that is at least 2-dimensional, with the first dimension corresponding to the time steps.

The targets could be a list or numpy array, where the first dimension should match with data. These are the target values that are aligned to the time steps of data. In some cases, data and targets could be the same.

The length argument controls the length of the samples generated by the TimeseriesGenerator in terms of the number of time steps.

In [4]:

# Create a TimeseriesGenerator object

from tensorflow.keras.preprocessing.sequence import TimeseriesGenerator

timeseries_gen = TimeseriesGenerator(dummy_data, dummy_targets, 4)

In [5]:

# Print the contents of the generator

print('Length:', len(timeseries_gen))
inputs, outputs = timeseries_gen[0]
print("\nData:")
print(inputs)
print("Targets:")
print(outputs)

Length: 1

Data:
[[1 2 3 4]
 [2 3 4 5]
 [3 4 5 6]
 [4 5 6 7]
 [5 6 7 8]
 [6 7 8 9]]
Targets:
[ 50  60  70  80  90 100]

We can see that the TimeseriesGenerator object has created inputs of length 4 from the data, and aligned them to the corresponding targets at the next time step. These inputs and targets have been batched together in numpy arrays.

Change the batch size¶

The TimeseriesGenerator also has a batch_size keyword argument. Let's create a generator with a batch size of 2. The default (maximum) batch size is 128.

In [6]:

# Create a TimeseriesGenerator object with length 3 and batch size 2

timeseries_gen = TimeseriesGenerator(dummy_data, dummy_targets, length=3, batch_size=2)

We can use the iter function to make our TimeseriesGenerator object iterable:

In [7]:

# Make the time series generator iterable

timeseries_iterator = iter(timeseries_gen)

Let's now generate some values using the iterator. Run the following cell a few times until the StopIteration of the generator object is reached. You will see that 2 samples are generated at a time until no more sample/target pairs can be formed:

In [8]:

# Iterate through the dataset examples

next(timeseries_iterator)

Out[8]:

(array([[1, 2, 3],
        [2, 3, 4]]),
 array([40, 50]))

You may have noticed that since there were 7 input/target sequences to generate in total, and our batch size was set to 2, the last batch consisted of just one input/target pair. Keep the possible difference in the size of the last batch in mind if you are using TimeseriesGenerator for applications that require a constant batch size.

Change the stride¶

The interval between consecutive samples can be adjusted using the stride keyword argument.

In [9]:

# Create a TimeseriesGenerator object with a stride of 2

timeseries_gen = TimeseriesGenerator(dummy_data, dummy_targets, length=3, stride=2, batch_size=1)

Above we have specified the length as 3, and the stride as 2. This means that we will generate sequences starting with the first sample being (1, 2, 3) to predict the target 40. Subsequent samples will each skip 2 timesteps since the stride is 2, meaning the next sample will will be (3, 4, 5) to predict 60, and the one after that will be (5, 6, 7) to predict 80.

Had we specified the stride to be the same as the length, then we would have samples which would not overlap, i.e. the first sequence would be the same, but the second sample would be (4, 5, 6) to predict 70, and so on.

In [10]:

# Make the time series generator iterable

timeseries_iterator = iter(timeseries_gen)

In [11]:

# Iterate through the dataset examples

while True:
    try:
        print(next(timeseries_iterator))
    except StopIteration:
        break

(array([[1, 2, 3]]), array([40]))
(array([[3, 4, 5]]), array([60]))
(array([[5, 6, 7]]), array([80]))
(array([[7, 8, 9]]), array([100]))

Reverse the time series¶

The reverse keyword argument will reverse the samples' output order. However, it will not change anything about the targets. That is, although the timesteps themselves will be reversed, the target will still be the timestep immediately following the end of the sample sequence.

In [12]:

# Create a reversed TimeseriesGenerator object

timeseries_gen = TimeseriesGenerator(dummy_data, dummy_targets, length=3, stride=1, batch_size=1, reverse=True)
timeseries_iterator = iter(timeseries_gen)

Let's generate a few samples and targets to see this in action:

In [13]:

# Iterate through the dataset examples

while True:
    try:
        print(next(timeseries_iterator))
    except StopIteration:
        break

(array([[3, 2, 1]]), array([40]))
(array([[4, 3, 2]]), array([50]))
(array([[5, 4, 3]]), array([60]))
(array([[6, 5, 4]]), array([70]))
(array([[7, 6, 5]]), array([80]))
(array([[8, 7, 6]]), array([90]))
(array([[9, 8, 7]]), array([100]))

Preprocess the DSD100 dataset¶

Load the audio files¶

We first need to load a track as an array that can be passed to the TimeseriesGenerator. To do this we can use the scipy package.

In [14]:

from scipy.io.wavfile import read, write

rate, song = read("data/055 - Angels In Amplifiers - I'm Alright/mixture.wav")
print("rate:", rate)
song = np.array(song)
print("song.shape:", song.shape)

rate: 44100
song.shape: (1942186, 2)

/home/bacti/.conda/envs/tensor/lib/python3.7/site-packages/ipykernel_launcher.py:3: WavFileWarning: Chunk (non-data) not understood, skipping it.
  This is separate from the ipykernel package so we can avoid doing imports until

The song is a stereo signal, which is why the second dimension here is equal to 2. The rate is the sample rate of the audio file. This number sets the speed at which the file should be played, so this audio runs at 44,100 samples per second (Hz).

Create a generator for the audio time series¶

Now that we know how to load the audio files, we can experiment with using them with the TimeseriesGenerator.

We'll specify the length of the sequences as 200,000, and we'll also set the stride to be 200,000. Remember that the audio runs at 44.1kHz, so 200,000 samples corresponds to about 4.5 seconds of audio.

Setting the length equal to the stride means that the different sequences will have no overlap. We will specify the batch size as 1 such that only one sample is generated at a time.

In [15]:

# Create a time series generator for the audio file

timeseries_gen = TimeseriesGenerator(song, targets=song, length=200000, stride=200000, batch_size=1)
timeseries_iterator = iter(timeseries_gen)

Running the cell below will generate 3 sequential samples from the iterator. If you wish to generate another 3 samples, you can run the cell again.

When run for the first time, the first sequence encompasses timesteps 1 - 200,000, the second will encompass items 200,001 - 400,000, and the third 400,001 - 600,000.

For each sequence, the target is the timestep immediately after the end of the sequence, and in this case (since we have the length equal to the stride) it will also be the first timestep of the following sample.

We will generate samples and write them as wav files. You can see in each case that the generated sample is a sequential chunk from the original audio. If played one after another, they'll form a continuous section of the track.

In [16]:

# Get three samples from the audio time series generator

for i in range(3):
    sample, target = next(timeseries_iterator)
    write('example.wav', rate, sample[0])
    print('Sample {}'.format(i+1))
    ipd.display(ipd.Audio("example.wav"))

Sample 1

Sample 2

Sample 3

Change the stride¶

In [17]:

# Create a TimeseriesGenerator object with the stride equal to half the length

timeseries_gen = TimeseriesGenerator(song, targets=song, length=200000, stride=100000, batch_size=1)
timeseries_iterator = iter(timeseries_gen)

With the stride equal to half of the length, we see that the samples are no longer non-overlapping. Each subsequent sample starts halfway through the previous sample.

In [18]:

# Get three samples from the audio time series generator

for i in range(3):
    sample, target = next(timeseries_iterator)
    write('example.wav', rate, sample[0])
    print('Sample {}'.format(i+1))
    ipd.display(ipd.Audio("example.wav"))

Sample 1

Sample 2

Sample 3

Change the sampling rate¶

Using this keyword argument results in downsampling. Increasing the sampling_rate from 1 to 2 will result in the interval between subsequent timesteps within any one sample being increased to 2. What this means is that only every other timestep in a given sample will be included.

Note that the length argument in this case refers to the length before the sampling_rate is applied.

In [19]:

# Create a TimeseriesGenerator object with sampling_rate set to 2

timeseries_gen = TimeseriesGenerator(song, song, length=200000, stride=200000, batch_size=1, sampling_rate=2)
timeseries_iterator = iter(timeseries_gen)

In the following, we will write the files with the original rate of 44,100Hz.

In [20]:

# Get three samples from the audio time series generator

for i in range(3):
    sample, target = next(timeseries_iterator)
    write('example.wav', rate, sample[0])
    print('Sample {}'.format(i+1))
    ipd.display(ipd.Audio("example.wav"))

Sample 1

Sample 2

Sample 3

The above clips each contain only 100,000 samples due to the sampling_rate being set to 2. This results in the audio files sounding twice as fast.

However, we could also adjust the rate at which we write the wav files to compensate for the downsampling. This results in clips that sound similar to the originals, but at a reduced quality.

If you are interested in audio signal processing, you may have noticed that since we did not filter out high frequency components before downsampling, we may have introduced aliasing.

In [21]:

# Write the wav files at an adjusted sample rate

for i in range(3):
    sample, target = next(timeseries_iterator)
    write('example.wav', rate//2, sample[0])
    print('Sample {}'.format(i+1))
    ipd.display(ipd.Audio("example.wav"))

Sample 1

Sample 2

Sample 3

Change the start index¶

The TimeseriesGenerator also has start_index and end_index keyword arguments, to specify which portion of our data we want to use to generate samples. This can be useful in the case that we want to reserve part of our data for validation.

Here we specify the start_index as 400,000 which is double the length, and will effectively skip the first 2 samples that would have otherwise been generated.

In [22]:

# Create a TimeseriesGenerator object with start index set to 400,000

timeseries_gen = TimeseriesGenerator(song, song, length=200000, stride=200000, batch_size=1, start_index=400000)
timeseries_iterator = iter(timeseries_gen)

In [23]:

# Get three samples from the audio time series generator

for i in range(3):
    sample, target = next(timeseries_iterator)
    write('example.wav', rate, sample[0])
    print('Sample {}'.format(i+1))
    ipd.display(ipd.Audio("example.wav"))

Sample 1

Sample 2

Sample 3

Shuffle the samples¶

Setting the keyword argument shuffle to True will randomly re-order the generated samples.

In [24]:

# Create a shuffled TimeseriesGenerator object

timeseries_gen = TimeseriesGenerator(song, song, length=200000, stride=200000, batch_size=1, shuffle=True)
timeseries_iterator = iter(timeseries_gen)

In [25]:

# Get three samples from the audio time series generator

for i in range(3):
    sample, target = next(timeseries_iterator)
    write('example.wav', rate, sample[0])
    print('Sample {}'.format(i+1))
    ipd.display(ipd.Audio("example.wav"))

Sample 1

Sample 2

Sample 3

Reverse the audio¶

As a final bit of fun, let's reverse some of the samples to see what they sound like backwards! Remember that the keyword argument reverse reverses the sample ordering of the timesteps within the sample.

In [26]:

# Create a reversed TimeseriesGenerator object

timeseries_gen = TimeseriesGenerator(song, song, length=200000, stride=200000, batch_size=1, reverse=True)
timeseries_iterator = iter(timeseries_gen)

Play the samples below to hear some slightly demonic-sounding music:

In [27]:

# Get three samples from the audio time series generator

for i in range(3):
    sample, target = next(timeseries_iterator)
    write('example.wav', rate, sample[0])
    print('Sample {}'.format(i+1))
    ipd.display(ipd.Audio("example.wav"))

Sample 1

Sample 2

Sample 3

Further reading and resources¶

• https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/TimeseriesGenerator
• https://keras.io/preprocessing/sequence/

Creating Datasets From Different Sources

In this reading notebook, we will explore a few of the ways in which we can load data into a tf.data.Dataset object.

In [1]:

import tensorflow as tf
print(tf.__version__)

2.2.0

The from_tensor_slices and from_tensors methods¶

We will start by looking at the from_tensor_slices and the from_tensors methods.

Both static methods are used to create datasets from Tensors or Tensor-like objects, such as numpy arrays or python lists. We can also pass in tuples and dicts of arrays or lists. The main distinction between the from_tensor_slices function and the from_tensors function is that the from_tensor_slices method will interpret the first dimension of the input data as the number of elements in the dataset, whereas the from_tensors method always results in a Dataset with a single element, containing the Tensor or tuple of Tensors passed.

In [2]:

# Create a random tensor with shape (3, 2)

example_tensor = tf.random.uniform([3,2])
print(example_tensor.shape)

(3, 2)

In [3]:

# Create two Datasets, using each static method

dataset1 = tf.data.Dataset.from_tensor_slices(example_tensor)
dataset2 = tf.data.Dataset.from_tensors(example_tensor)

In [4]:

# Print the element_spec for each

print(dataset1.element_spec)
print(dataset2.element_spec)

TensorSpec(shape=(2,), dtype=tf.float32, name=None)
TensorSpec(shape=(3, 2), dtype=tf.float32, name=None)

As seen above, creating the Dataset using the from_tensor_slices method slices the given array or Tensor along the first dimension to produce a set of elements for the Dataset.

This means that although we could pass any Tensor - or tuple of Tensors - to the from_tensors method, the same cannot be said of the from_tensor_slices method, which has the additional requirement that each Tensor in the list has the same size in the zeroth dimension.

In [5]:

# Create three Tensors with different shapes

tensor1 = tf.random.uniform([10,2,2])
tensor2 = tf.random.uniform([10,1])
tensor3 = tf.random.uniform([9,2,2])

We cannot create a Dataset using the from_tensor_slices method from a list of tensor1 and tensor3 since they do not have the same size in the first dimension:

In [6]:

# Try to create a Dataset from tensor1 and tensor3 using from_tensor_slices - this will raise an error

dataset = tf.data.Dataset.from_tensor_slices((tensor1, tensor3))

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-8d158a73c9fd> in <module>
      1 # Try to create a Dataset from tensor1 and tensor3 using from_tensor_slices - this will raise an error
      2 
----> 3 dataset = tf.data.Dataset.from_tensor_slices((tensor1, tensor3))

~/.conda/envs/tensor/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py in from_tensor_slices(tensors)
    638       Dataset: A `Dataset`.
    639     """
--> 640     return TensorSliceDataset(tensors)
    641 
    642   class _GeneratorState(object):

~/.conda/envs/tensor/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py in __init__(self, element)
   2866     for t in self._tensors[1:]:
   2867       batch_dim.assert_is_compatible_with(tensor_shape.Dimension(
-> 2868           tensor_shape.dimension_value(t.get_shape()[0])))
   2869 
   2870     variant_tensor = gen_dataset_ops.tensor_slice_dataset(

~/.conda/envs/tensor/lib/python3.7/site-packages/tensorflow/python/framework/tensor_shape.py in assert_is_compatible_with(self, other)
    274     if not self.is_compatible_with(other):
    275       raise ValueError("Dimensions %s and %s are not compatible" %
--> 276                        (self, other))
    277 
    278   def merge_with(self, other):

ValueError: Dimensions 10 and 9 are not compatible

However, we can of course create a Dataset from this tuple using the from_tensors method, which interprets the tuple as a single element.

In [7]:

# Create a Dataset from tensor1 and tensor3 using from_tensors

dataset = tf.data.Dataset.from_tensors((tensor1, tensor3))
dataset.element_spec

Out[7]:

(TensorSpec(shape=(10, 2, 2), dtype=tf.float32, name=None),
 TensorSpec(shape=(9, 2, 2), dtype=tf.float32, name=None))

Although tensor1 and tensor2 do not have the same shape, or even same rank (number of dimensions), we can still use the from_tensor_slices method to form a dataset from a list of these tensors, since they have the same size in the first dimension.

In [8]:

# Create a Dataset from tensor1 and tensor2

dataset = tf.data.Dataset.from_tensor_slices((tensor1, tensor2))
dataset.element_spec

Out[8]:

(TensorSpec(shape=(2, 2), dtype=tf.float32, name=None),
 TensorSpec(shape=(1,), dtype=tf.float32, name=None))

In the above, the first dimension was interpreted as the number of elements in the Dataset, as expected.

Creating Datasets from numpy arrays¶

We can also use the from_tensor_slices and from_tensors methods to create Datasets from numpy arrays. In fact, behind the scenes, the numpy array is converted to a set of tf.constant operations to populate the Tensor in the TensorFlow graph.

In [9]:

# Create a numpy array dataset

import numpy as np

numpy_array = np.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]], [[9, 10], [11, 12]]])
print(numpy_array.shape)

(3, 2, 2)

In [10]:

# Create two Datasets, using each static method

dataset1 = tf.data.Dataset.from_tensor_slices(numpy_array)
dataset2 = tf.data.Dataset.from_tensors(numpy_array)

print(dataset1.element_spec)
print(dataset2.element_spec)

TensorSpec(shape=(2, 2), dtype=tf.int64, name=None)
TensorSpec(shape=(3, 2, 2), dtype=tf.int64, name=None)

As before, from_tensors interprets the entire array as a single element, whereas from_tensor_slices slices the array along the first dimension to form the elements.

Creating Datasets from pandas DataFrames¶

A pandas DataFrame can be easily converted to a Dataset using the from_tensor_slices method.

The Balloons dataset¶

A pandas DataFrame can be loaded from a CSV file. We will use the Balloons dataset to demonstrate. This dataset is stored in a CSV file, and contains a list of attributes describing instances of a balloon inflation experiment, such as the colour and size of the balloon, the age of the person who performed the attempted inflation, and the way in which they did it. Finally, there is the target column "Inflated", which is either T for True, or F for False, indicating whether or not the person managed to inflate the balloon.

In [11]:

# Load the CSV file into a Dataframe

import pandas as pd

pandas_dataframe = pd.read_csv('data/balloon_dataset.csv')

In [12]:

# Inspect the data

pandas_dataframe.head()

Out[12]:

	Colour	Size	Act	Age	Inflated
0	YELLOW	SMALL	STRETCH	ADULT	T
1	YELLOW	SMALL	STRETCH	ADULT	T
2	YELLOW	SMALL	STRETCH	CHILD	F
3	YELLOW	SMALL	DIP	ADULT	F
4	YELLOW	SMALL	DIP	CHILD	F

To convert the DataFrame to a Dataset, we first convert the DataFrame to a dictionary. By doing this, we preserve the column names as the dictionary labels.

Note: A Dataset can be formed from either a tuple or a dict of Tensors. We saw above a number of Datasets being formed from a tuple. The only distinction for a Dataset formed from a dict is that the Dataset items will be dicts accessed by key, rather than tuples accessed by index.

In [13]:

# Convert the DataFrame to a dict

dataframe_dict = dict(pandas_dataframe)
print(dataframe_dict.keys())

dict_keys(['Colour', 'Size', 'Act', 'Age', 'Inflated'])

We can now run the from_tensor_slices method on this dict and print the resulting Dataset element_spec, as well as an example element. Note that since we formed the Dataset from a dict, we see the column (dictionary) names in the element_spec.

In [14]:

# Create the Dataset

pandas_dataset = tf.data.Dataset.from_tensor_slices(dataframe_dict)

In [15]:

# View the Dataset element_spec

pandas_dataset.element_spec

Out[15]:

{'Colour': TensorSpec(shape=(), dtype=tf.string, name=None),
 'Size': TensorSpec(shape=(), dtype=tf.string, name=None),
 'Act': TensorSpec(shape=(), dtype=tf.string, name=None),
 'Age': TensorSpec(shape=(), dtype=tf.string, name=None),
 'Inflated': TensorSpec(shape=(), dtype=tf.string, name=None)}

In [16]:

# Iterate the Dataset

next(iter(pandas_dataset))

Out[16]:

{'Colour': <tf.Tensor: shape=(), dtype=string, numpy=b'YELLOW'>,
 'Size': <tf.Tensor: shape=(), dtype=string, numpy=b'SMALL'>,
 'Act': <tf.Tensor: shape=(), dtype=string, numpy=b'STRETCH'>,
 'Age': <tf.Tensor: shape=(), dtype=string, numpy=b'ADULT'>,
 'Inflated': <tf.Tensor: shape=(), dtype=string, numpy=b'T'>}

Creating Datasets directly from CSV Files¶

The TensorFlow experimental library contains a variety of functions and classes contributed by the community that may not be ready for release into the main TensorFlow library in their immediate form, but which may be included in TensorFlow in the future. One such useful experimental function is the tf.data.experimental.make_csv_dataset function. This allows us to read CSV data from the disk directly into a Dataset object.

We will run the function on the example CSV file from disk, and specify the batch size and the name of the target column, which is used to structure the Dataset into an (input, target) tuple.

Note: Because of the ephemeral nature of the experimental package, you may well get warnings printed in the console when using a function or class contained in the package for the first time.

In [17]:

# Create the Dataset from the CSV file

csv_dataset = tf.data.experimental.make_csv_dataset('data/balloon_dataset.csv',
                                                    batch_size=1,
                                                    label_name='Inflated')

To check that we've loaded our Dataset correctly, let's print the element_spec:

In [18]:

# View the Dataset element_spec

csv_dataset.element_spec

Out[18]:

(OrderedDict([('Colour', TensorSpec(shape=(1,), dtype=tf.string, name=None)),
              ('Size', TensorSpec(shape=(1,), dtype=tf.string, name=None)),
              ('Act', TensorSpec(shape=(1,), dtype=tf.string, name=None)),
              ('Age', TensorSpec(shape=(1,), dtype=tf.string, name=None))]),
 TensorSpec(shape=(1,), dtype=tf.string, name=None))

In [19]:

# Iterate the Dataset

next(iter(csv_dataset))

Out[19]:

(OrderedDict([('Colour',
               <tf.Tensor: shape=(1,), dtype=string, numpy=array([b'PURPLE'], dtype=object)>),
              ('Size',
               <tf.Tensor: shape=(1,), dtype=string, numpy=array([b'LARGE'], dtype=object)>),
              ('Act',
               <tf.Tensor: shape=(1,), dtype=string, numpy=array([b'DIP'], dtype=object)>),
              ('Age',
               <tf.Tensor: shape=(1,), dtype=string, numpy=array([b'ADULT'], dtype=object)>)]),
 <tf.Tensor: shape=(1,), dtype=string, numpy=array([b'F'], dtype=object)>)

Note that in the above Dataset, the target column Inflated does not have a key, since it is uniquely accessible as the second element of the tuple, whereas the attributes which reside as a dictionary of Tensors in the first element retain their labels so we can distinguish them.

Further reading and resources¶

• https://www.tensorflow.org/guide/data
• https://www.tensorflow.org/tutorials/load_data/csv
• https://www.tensorflow.org/tutorials/load_data/pandas_dataframe
• https://www.tensorflow.org/api_docs/python/tf/data/Dataset

TensorFlow Datasets

In this reading notebook, we will take a look at the tensorflow-datasets library.

We have previously made use of the tf.keras.datasets package, which gave us access to a variety of useful datasets such as the IMDB movie dataset, the CIFAR-100 small image classification dataset, and the MNIST handwritten digits dataset.

The tensorflow-datasets library gives us another means of accessing a variety of useful datasets.

Installation¶

The tensorflow-datasets library is installed independently of TensorFlow itself. It can be installed using pip, by running the following command in a terminal (assuming that tensorflow has already been installed):

pip install tensorflow-datasets

Listing the Available Datasets¶

The list_builders function can be used to list the available datasets.

In [1]:

# List available datasets
import tensorflow_datasets as tfds
tfds.list_builders()

Out[1]:

['abstract_reasoning',
 'aflw2k3d',
 'amazon_us_reviews',
 'bair_robot_pushing_small',
 'bigearthnet',
 'binarized_mnist',
 'binary_alpha_digits',
 'caltech101',
 'caltech_birds2010',
 'caltech_birds2011',
 'cats_vs_dogs',
 'celeb_a',
 'celeb_a_hq',
 'chexpert',
 'cifar10',
 'cifar100',
 'cifar10_corrupted',
 'clevr',
 'cnn_dailymail',
 'coco',
 'coco2014',
 'coil100',
 'colorectal_histology',
 'colorectal_histology_large',
 'curated_breast_imaging_ddsm',
 'cycle_gan',
 'deep_weeds',
 'definite_pronoun_resolution',
 'diabetic_retinopathy_detection',
 'downsampled_imagenet',
 'dsprites',
 'dtd',
 'dummy_dataset_shared_generator',
 'dummy_mnist',
 'emnist',
 'eurosat',
 'fashion_mnist',
 'flores',
 'food101',
 'gap',
 'glue',
 'groove',
 'higgs',
 'horses_or_humans',
 'image_label_folder',
 'imagenet2012',
 'imagenet2012_corrupted',
 'imdb_reviews',
 'iris',
 'kitti',
 'kmnist',
 'lfw',
 'lm1b',
 'lsun',
 'mnist',
 'mnist_corrupted',
 'moving_mnist',
 'multi_nli',
 'nsynth',
 'omniglot',
 'open_images_v4',
 'oxford_flowers102',
 'oxford_iiit_pet',
 'para_crawl',
 'patch_camelyon',
 'pet_finder',
 'quickdraw_bitmap',
 'resisc45',
 'rock_paper_scissors',
 'rock_you',
 'scene_parse150',
 'shapes3d',
 'smallnorb',
 'snli',
 'so2sat',
 'squad',
 'stanford_dogs',
 'stanford_online_products',
 'starcraft_video',
 'sun397',
 'super_glue',
 'svhn_cropped',
 'ted_hrlr_translate',
 'ted_multi_translate',
 'tf_flowers',
 'titanic',
 'trivia_qa',
 'uc_merced',
 'ucf101',
 'visual_domain_decathlon',
 'voc2007',
 'wikipedia',
 'wmt14_translate',
 'wmt15_translate',
 'wmt16_translate',
 'wmt17_translate',
 'wmt18_translate',
 'wmt19_translate',
 'wmt_t2t_translate',
 'wmt_translate',
 'xnli']

Loading a Dataset¶

Loading a particular dataset is straightforward; simply use the load function, specifying the name and any other keyword arguments. In the example code below, we demonstrate how to load the kmnist dataset using the function.

As stated on the documentation page, running the function with split=None (the default) returns a dictionary of the splits, with the keys test and train.

In [2]:

# Load the mnist_corrupted dataset
kmnist  = tfds.load(name = 'kmnist', split = None)
kmnist_train = kmnist['train']
kmnist_test = kmnist['test']

WARNING:absl:Warning: Setting shuffle_files=True because split=TRAIN and shuffle_files=None. This behavior will be deprecated on 2019-08-06, at which point shuffle_files=False will be the default for all splits.

In [3]:

# View some examples from the dataset

import matplotlib.pyplot as plt
%matplotlib inline
import tensorflow as tf

fig, axes = plt.subplots(3, 3, figsize = (8, 8))
fig.subplots_adjust(hspace = 0.2, wspace = 0.1)

for i, (elem, ax) in enumerate(zip(kmnist_train, axes.flat)):
    image = tf.squeeze(elem['image'])
    label = elem['label']
    
    ax.imshow(image, cmap = 'gray')
    ax.text (
        0.7, -0.12, f'Digit = {label}', ha = 'right',
        transform = ax.transAxes, color = 'black'
    )
    ax.set_xticks([])
    ax.set_yticks([])

Further reading and resources¶

• https://www.tensorflow.org/datasets
• https://www.tensorflow.org/datasets/catalog/overview
• https://wwww.tensorflow.org/datasets/api_docs/python/tfds
• https://tensorflow.org/datasets/api_docs/python/tfds/load

Data Pipeline

In [1]:

import tensorflow as tf
print(tf.__version__)

2.2.0

1. Keras datasets¶

2. Dataset generators¶

3. Keras image data augmentation¶

4. The Dataset class¶

5. Training with Datasets¶

---

Keras datasets¶

For a list of Keras datasets and documentation on recommended usage, see this link.

In [2]:

import numpy as np
import matplotlib.pyplot as plt

Load the CIFAR-100 Dataset¶

In [3]:

from tensorflow.keras.datasets import cifar100

In [4]:

# Load the CIFAR-100 dataset
(train_images, train_labels), (test_images, test_labels) = cifar100.load_data(label_mode = 'fine')

Downloading data from https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz
169009152/169001437 [==============================] - 343s 2us/step

In [5]:

# Confirm that reloading the dataset does not require a download
(train_images, train_labels), (test_images, test_labels) = cifar100.load_data(label_mode = 'fine')

Examine the Dataset¶

In [6]:

# Examine the shape of the data.
print(test_images.shape)
print(test_labels.shape)

(10000, 32, 32, 3)
(10000, 1)

In [7]:

# Examine one of the images and its corresponding label
plt.imshow(train_images[500])
print(train_labels[500])

[41]

In [8]:

# Load the list of labels from a JSON file
import json
with open('data/cifar100_fine_labels.json', 'r') as fine_labels:
    cifar100_fine_labels = json.load(fine_labels)

The list of labels for the CIFAR-100 dataset are available here.

In [9]:

# Print a few of the labels
cifar100_fine_labels[0:10]

Out[9]:

['apple',
 'aquarium_fish',
 'baby',
 'bear',
 'beaver',
 'bed',
 'bee',
 'beetle',
 'bicycle',
 'bottle']

In [10]:

# Print the corresponding label for the example above
cifar100_fine_labels[41]

Out[10]:

'lawn_mower'

Load the data using different label modes¶

In [11]:

# Display a few examples from category 87 (index 86) and the list of labels
examples = train_images[(train_labels.T == 86)[0]][:3]
fig, ax = plt.subplots(1,3)
ax[0].imshow(examples[0])
ax[1].imshow(examples[1])
ax[2].imshow(examples[2])

Out[11]:

<matplotlib.image.AxesImage at 0x7fad15f9c990>

In [12]:

# Reload the data using the 'coarse' label mode
(train_images, train_labels), (test_images, test_labels) = cifar100.load_data(label_mode = 'coarse')

In [13]:

# Display three images from the dataset with the label 6 (index 5)
examples = train_images[(train_labels.T == 5)[0]][:3]
fig, ax = plt.subplots(1,3)
ax[0].imshow(examples[0])
ax[1].imshow(examples[1])
ax[2].imshow(examples[2])

Out[13]:

<matplotlib.image.AxesImage at 0x7fad15e787d0>

In [14]:

# Load the list of coarse labels from a JSON file
with open('data/cifar100_coarse_labels.json', 'r') as coarse_labels:
    cifar100_coarse_labels = json.load(coarse_labels)

In [15]:

# Print a few of the labels
cifar100_coarse_labels[0:10]

Out[15]:

['aquatic mammals',
 'fish',
 'flowers',
 'food containers',
 'fruit and vegetables',
 'household electrical device',
 'household furniture',
 'insects',
 'large carnivores',
 'large man-made outdoor things']

In [16]:

# Print the corresponding label for the example above
cifar100_fine_labels[86], cifar100_coarse_labels[5]

Out[16]:

('telephone', 'household electrical device')

Load the IMDB Dataset¶

In [17]:

from tensorflow.keras.datasets import imdb

In [18]:

# Load the IMDB dataset
(train_data, train_labels), (test_data, test_labels) = imdb.load_data()

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz
17465344/17464789 [==============================] - 12s 1us/step

/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/imdb.py:155: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/imdb.py:156: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])

In [19]:

# Print an example from the training dataset, along with its corresponding label
print(train_data[0])
print(train_labels[0])

[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 22665, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 21631, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 19193, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 10311, 8, 4, 107, 117, 5952, 15, 256, 4, 31050, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 12118, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]
1

In [20]:

# Get the lengths of the input sequences
sequence_lengths = [len(seq) for seq in train_data]

In [21]:

# Determine the maximum and minimum sequence length
print(np.max(sequence_lengths))
print(np.min(sequence_lengths))

2494
11

Using Keyword Arguments¶

In [22]:

# Load the data ignoring the 50 most frequent words, use oov_char=2 (this is the default)
(train_data, train_labels), (test_data, test_labels) = imdb.load_data(skip_top = 50, oov_char = 2)

In [23]:

# Get the lengths of the input sequences
sequence_lengths = [len(seq) for seq in train_data]

In [24]:

# Determine the maximum and minimum sequence length
print(np.max(sequence_lengths))
print(np.min(sequence_lengths))

2494
11

In [25]:

# Define functions for filtering the sequences

def remove_oov_char(element):
    ''' Filter function for removing the oov_char. '''
    return [word for word in element if word !=2 ]

def filter_list(lst):
    ''' Run remove_oov_char on elements in a list. '''
    return [remove_oov_char(element) for element in lst]

In [26]:

# Remove the oov_char from the sequences using the filter_list function
train_data = filter_list(train_data)

In [27]:

# Get the lengths of the input sequences
sequence_lengths = [len(seq) for seq in train_data]

In [28]:

# Determine the maximum and minimum sequence length
print(np.max(sequence_lengths))
print(np.min(sequence_lengths))

1648
3

---

Dataset generators¶

In [29]:

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

Load the UCI Fertility Dataset¶

We will be using a dataset available at https://archive.ics.uci.edu/ml/datasets/Fertility from UC Irvine.

In [30]:

# Load the fertility dataset
headers = [
    'Season', 'Age', 'Diseases', 'Trauma', 'Surgery', 'Fever',
    'Alcohol', 'Smoking', 'Sitting', 'Output'
]
fertility = pd.read_csv('data/fertility_diagnosis.txt', delimiter = ',', header = None, names = headers)

In [31]:

# Print the shape of the DataFrame
print(fertility.shape)

(100, 10)

In [32]:

# Show the head of the DataFrame
fertility.head()

Out[32]:

	Season	Age	Diseases	Trauma	Surgery	Fever	Alcohol	Smoking	Sitting	Output
0	-0.33	0.69	0	1	1	0	0.8	0	0.88	N
1	-0.33	0.94	1	0	1	0	0.8	1	0.31	O
2	-0.33	0.50	1	0	0	0	1.0	-1	0.50	N
3	-0.33	0.75	0	1	1	0	1.0	-1	0.38	N
4	-0.33	0.67	1	1	0	0	0.8	-1	0.50	O

Process the data¶

In [33]:

# Map the 'Output' feature from 'N' to 0 and from 'O' to 1
fertility['Output'] = fertility['Output'].map(lambda x : 0.0 if x == 'N' else 1.0)

In [34]:

# Show the head of the DataFrame
fertility.head()

Out[34]:

	Season	Age	Diseases	Trauma	Surgery	Fever	Alcohol	Smoking	Sitting	Output
0	-0.33	0.69	0	1	1	0	0.8	0	0.88	0.0
1	-0.33	0.94	1	0	1	0	0.8	1	0.31	1.0
2	-0.33	0.50	1	0	0	0	1.0	-1	0.50	0.0
3	-0.33	0.75	0	1	1	0	1.0	-1	0.38	0.0
4	-0.33	0.67	1	1	0	0	0.8	-1	0.50	1.0

In [35]:

# Convert the DataFrame so that the features are mapped to floats
fertility = fertility.astype('float32')

In [36]:

# Shuffle the DataFrame
fertility = fertility.sample(frac = 1).reset_index(drop = True)

In [37]:

# Show the head of the DataFrame
fertility.head()

Out[37]:

	Season	Age	Diseases	Trauma	Surgery	Fever	Alcohol	Smoking	Sitting	Output
0	-0.33	0.53	1.0	1.0	0.0	0.0	0.8	0.0	0.75	0.0
1	1.00	0.58	0.0	0.0	0.0	1.0	0.8	1.0	0.44	0.0
2	-1.00	0.72	1.0	1.0	1.0	1.0	0.8	-1.0	0.19	0.0
3	1.00	0.67	1.0	0.0	0.0	0.0	1.0	-1.0	0.25	0.0
4	-0.33	0.81	1.0	1.0	1.0	0.0	1.0	1.0	0.38	0.0

In [38]:

# Convert the field Season to a one-hot encoded vector
fertility = pd.get_dummies(fertility, prefix = 'Season', columns = ['Season'])

In [39]:

# Show the head of the DataFrame
fertility.head()

Out[39]:

	Age	Diseases	Trauma	Surgery	Fever	Alcohol	Smoking	Sitting	Output	Season_-1.0	Season_-0.33000001311302185	Season_0.33000001311302185	Season_1.0
0	0.53	1.0	1.0	0.0	0.0	0.8	0.0	0.75	0.0	0	1	0	0
1	0.58	0.0	0.0	0.0	1.0	0.8	1.0	0.44	0.0	0	0	0	1
2	0.72	1.0	1.0	1.0	1.0	0.8	-1.0	0.19	0.0	1	0	0	0
3	0.67	1.0	0.0	0.0	0.0	1.0	-1.0	0.25	0.0	0	0	0	1
4	0.81	1.0	1.0	1.0	0.0	1.0	1.0	0.38	0.0	0	1	0	0

In [40]:

# Move the Output column such that it is the last column in the DataFrame
fertility.columns = [col for col in fertility.columns if col != 'Output'] + ['Output']

In [41]:

# Show the head of the DataFrame
fertility.head()

Out[41]:

	Age	Diseases	Trauma	Surgery	Fever	Alcohol	Smoking	Sitting	Season_-1.0	Season_-0.33000001311302185	Season_0.33000001311302185	Season_1.0	Output
0	0.53	1.0	1.0	0.0	0.0	0.8	0.0	0.75	0.0	0	1	0	0
1	0.58	0.0	0.0	0.0	1.0	0.8	1.0	0.44	0.0	0	0	0	1
2	0.72	1.0	1.0	1.0	1.0	0.8	-1.0	0.19	0.0	1	0	0	0
3	0.67	1.0	0.0	0.0	0.0	1.0	-1.0	0.25	0.0	0	0	0	1
4	0.81	1.0	1.0	1.0	0.0	1.0	1.0	0.38	0.0	0	1	0	0

In [42]:

# Convert the DataFrame to a numpy array.
fertility = fertility.to_numpy()

Split the Data¶

In [43]:

# Split the dataset into training and validation set
training = fertility[0:70]
validation = fertility[70:100]

In [44]:

# Verify the shape of the training data
training.shape

Out[44]:

(70, 13)

In [45]:

# Separate the features and labels for the validation and training data
training_features = training[:,0:-1]
training_labels = training[:,-1]
validation_features = validation[:,0:-1]
validation_labels = validation[:,-1]

Create the Generator¶

In [46]:

# Create a function that returns a generator producing inputs and labels
def get_generator(features, labels, batch_size = 1):
    for n in range(int(len(features) / batch_size)):
        yield (features[n * batch_size: (n + 1) * batch_size], labels[n * batch_size: (n + 1) * batch_size])

In [47]:

# Apply the function to our training features and labels with a batch size of 10
train_generator = get_generator(training_features, training_labels, batch_size = 10)

In [48]:

# Test the generator using the next() function
next(train_generator)

Out[48]:

(array([[ 0.53,  1.  ,  1.  ,  0.  ,  0.  ,  0.8 ,  0.  ,  0.75,  0.  ,
          0.  ,  1.  ,  0.  ],
        [ 0.58,  0.  ,  0.  ,  0.  ,  1.  ,  0.8 ,  1.  ,  0.44,  0.  ,
          0.  ,  0.  ,  0.  ],
        [ 0.72,  1.  ,  1.  ,  1.  ,  1.  ,  0.8 , -1.  ,  0.19,  0.  ,
          1.  ,  0.  ,  0.  ],
        [ 0.67,  1.  ,  0.  ,  0.  ,  0.  ,  1.  , -1.  ,  0.25,  0.  ,
          0.  ,  0.  ,  0.  ],
        [ 0.81,  1.  ,  1.  ,  1.  ,  0.  ,  1.  ,  1.  ,  0.38,  0.  ,
          0.  ,  1.  ,  0.  ],
        [ 0.92,  1.  ,  0.  ,  0.  ,  1.  ,  0.6 , -1.  ,  0.19,  0.  ,
          0.  ,  1.  ,  0.  ],
        [ 0.78,  1.  ,  1.  ,  1.  ,  0.  ,  0.6 ,  0.  ,  0.13,  0.  ,
          0.  ,  0.  ,  0.  ],
        [ 0.64,  1.  ,  1.  ,  1.  ,  0.  ,  0.8 , -1.  ,  0.31,  0.  ,
          0.  ,  1.  ,  0.  ],
        [ 0.78,  1.  ,  0.  ,  1.  ,  1.  ,  1.  ,  1.  ,  0.25,  0.  ,
          1.  ,  0.  ,  0.  ],
        [ 0.75,  1.  ,  1.  ,  1.  ,  0.  ,  1.  ,  1.  ,  0.25,  0.  ,
          0.  ,  0.  ,  0.  ]], dtype=float32),
 array([0., 1., 0., 1., 0., 0., 1., 0., 0., 1.], dtype=float32))

Build the model¶

In [49]:

# Create a model using Keras with 3 layers

from tensorflow.keras import Model
from tensorflow.keras.layers import Dense, Input, BatchNormalization

input_shape = (12,)
output_shape = (1,)

model_input = Input(input_shape)
batch_1 = BatchNormalization(momentum = 0.8)(model_input)
dense_1 = Dense(100, activation = 'relu')(batch_1)
batch_2 = BatchNormalization(momentum = 0.8)(dense_1)
output = Dense(1, activation = 'sigmoid')(batch_2)

model = Model([model_input], output)

In [50]:

# Display the model summary to show the resultant structure
model.summary()

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 12)]              0         
_________________________________________________________________
batch_normalization (BatchNo (None, 12)                48        
_________________________________________________________________
dense (Dense)                (None, 100)               1300      
_________________________________________________________________
batch_normalization_1 (Batch (None, 100)               400       
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 101       
=================================================================
Total params: 1,849
Trainable params: 1,625
Non-trainable params: 224
_________________________________________________________________

Compile the model¶

In [51]:

# Create the optimizer object
optimizer = tf.keras.optimizers.Adam(learning_rate = 1e-2)

In [52]:

# Compile the model with loss function and metric
model.compile(optimizer = optimizer, loss = 'binary_crossentropy', metrics = ['accuracy'])

Train and evaluate the model using the generator¶

In [53]:

# Calculate the number of training steps per epoch for the given batch size.
batch_size = 5
train_steps = len(training) // batch_size

In [54]:

# Set the epochs to 3
epochs = 3

In [55]:

# Train the model
for epoch in range(epochs):
    train_generator = get_generator(training_features, training_labels, batch_size = batch_size)
    validation_generator = get_generator(validation_features, validation_labels, batch_size = 30)
    model.fit_generator (
        train_generator, steps_per_epoch = train_steps,
        validation_data = validation_generator, validation_steps = 1
    )

WARNING:tensorflow:From <ipython-input-55-f8502d3670c5>:7: Model.fit_generator (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
Please use Model.fit, which supports generators.
14/14 [==============================] - 0s 14ms/step - loss: 0.6381 - accuracy: 0.7143 - val_loss: 0.2039 - val_accuracy: 0.9000
14/14 [==============================] - 0s 5ms/step - loss: 0.2201 - accuracy: 0.9000 - val_loss: 0.0471 - val_accuracy: 1.0000
14/14 [==============================] - 0s 4ms/step - loss: 0.1614 - accuracy: 0.9429 - val_loss: 0.0208 - val_accuracy: 1.0000

In [56]:

# Try to run the fit_generator function once more; observe what happens
model.fit_generator(train_generator, steps_per_epoch = train_steps)

---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-56-867a16199ae9> in <module>
      1 # Try to run the fit_generator function once more; observe what happens
----> 2 model.fit_generator(train_generator, steps_per_epoch = train_steps)

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py in new_func(*args, **kwargs)
    322               'in a future version' if date is None else ('after %s' % date),
    323               instructions)
--> 324       return func(*args, **kwargs)
    325     return tf_decorator.make_decorator(
    326         func, new_func, 'deprecated',

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, validation_freq, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
   1477         use_multiprocessing=use_multiprocessing,
   1478         shuffle=shuffle,
-> 1479         initial_epoch=initial_epoch)
   1480 
   1481   @deprecation.deprecated(

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
     64   def _method_wrapper(self, *args, **kwargs):
     65     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
---> 66       return method(self, *args, **kwargs)
     67 
     68     # Running inside `run_distribute_coordinator` already.

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
    813           workers=workers,
    814           use_multiprocessing=use_multiprocessing,
--> 815           model=self)
    816 
    817       # Container that configures and calls `tf.keras.Callback`s.

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py in __init__(self, x, y, sample_weight, batch_size, steps_per_epoch, initial_epoch, epochs, shuffle, class_weight, max_queue_size, workers, use_multiprocessing, model)
   1110         use_multiprocessing=use_multiprocessing,
   1111         distribution_strategy=ds_context.get_strategy(),
-> 1112         model=model)
   1113 
   1114     strategy = ds_context.get_strategy()

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py in __init__(self, x, y, sample_weights, workers, use_multiprocessing, max_queue_size, model, **kwargs)
    770     # Since we have to know the dtype of the python generator when we build the
    771     # dataset, we have to look at a batch to infer the structure.
--> 772     peek, x = self._peek_and_restore(x)
    773     assert_not_namedtuple(peek)
    774     peek = self._standardize_batch(peek)

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py in _peek_and_restore(x)
    828   @staticmethod
    829   def _peek_and_restore(x):
--> 830     peek = next(x)
    831     return peek, itertools.chain([peek], x)
    832 

StopIteration: 

Make an infinitely looping generator¶

In [57]:

# Create a function that returns an infinitely looping generator
def get_generator_cyclic(features, labels, batch_size = 1):
    while True:
        for n in range(int(len(features) / batch_size)):
            yield (
                features[n * batch_size: (n + 1) * batch_size],
                labels[n * batch_size: (n + 1) * batch_size]
            )
        permuted = np.random.permutation(len(features))
        features = features[permuted]
        labels = labels[permuted]

In [58]:

# Create a generator using this function.
train_generator_cyclic = get_generator_cyclic(training_features, training_labels, batch_size = batch_size)

In [59]:

# Assert that the new cyclic generator does not raise a StopIteration
for i in range(2 * train_steps):
    next(train_generator_cyclic)

In [60]:

# Generate a cyclic validation generator
validation_generator_cyclic = get_generator_cyclic (
    validation_features, validation_labels, batch_size = batch_size
)

In [61]:

# Train the model
model.fit_generator (
    train_generator_cyclic, steps_per_epoch = train_steps,
    validation_data = validation_generator_cyclic, validation_steps = 1,
    epochs = 3
)

Epoch 1/3
14/14 [==============================] - 0s 4ms/step - loss: 0.1046 - accuracy: 1.0000 - val_loss: 0.0032 - val_accuracy: 1.0000
Epoch 2/3
14/14 [==============================] - 0s 4ms/step - loss: 0.1394 - accuracy: 0.9714 - val_loss: 7.1643e-04 - val_accuracy: 1.0000
Epoch 3/3
14/14 [==============================] - 0s 4ms/step - loss: 0.0188 - accuracy: 1.0000 - val_loss: 0.0011 - val_accuracy: 1.0000

Out[61]:

<tensorflow.python.keras.callbacks.History at 0x7fad0d3cd350>

Evaluate the model and get predictions¶

In [62]:

# Let's obtain a validation data generator.
validation_generator = get_generator(validation_features, validation_labels, batch_size = 30)

In [63]:

# Get predictions on the validation data
predictions = model.predict_generator(validation_generator, steps = 1)
print(np.round(predictions.T[0]))

WARNING:tensorflow:From <ipython-input-63-c25ca45d9cef>:2: Model.predict_generator (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
Please use Model.predict, which supports generators.
[0. 0. 0. 0. 0. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 1. 0. 0. 1.]

In [64]:

# Print the corresponding validation labels
print(validation_labels)

[0. 0. 0. 0. 0. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 1. 0. 0. 1.]

In [65]:

# Obtain a validation data generator
validation_generator = get_generator(validation_features, validation_labels, batch_size = 30)

In [66]:

# Evaluate the model
print(model.evaluate(validation_generator))

1/1 [==============================] - 0s 9ms/step - loss: 0.0012 - accuracy: 1.0000
[0.0011701274197548628, 1.0]

---

Keras image data augmentation¶

In [67]:

import matplotlib.pyplot as plt
import numpy as np

Load the CIFAR-10 Dataset¶

In [68]:

from tensorflow.keras.datasets import cifar10

In [69]:

# Load the CIFAR-10 dataset
(training_features, training_labels), (test_features, test_labels) = cifar10.load_data()

Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
170500096/170498071 [==============================] - 427s 3us/step

In [70]:

# Convert the labels to a one-hot encoding
num_classes = 10
training_labels = tf.keras.utils.to_categorical(training_labels, num_classes)
test_labels = tf.keras.utils.to_categorical(test_labels, num_classes)

Create a generator function¶

In [71]:

# Create a function that returns a data generator
def get_generator(features, labels, batch_size = 1):
    for n in range(int(len(features) / batch_size)):
        yield (features[n * batch_size: (n + 1) * batch_size], labels[n * batch_size: (n + 1) * batch_size])

In [72]:

# Use the function we created to get a training data generator with a batch size of 1
training_generator = get_generator(training_features, training_labels)

In [73]:

# Assess the shape of the items generated by training_generator using the `next` function to yield an item.
image, label = next(training_generator)
print(image.shape)
print(label.shape)

(1, 32, 32, 3)
(1, 10)

In [74]:

# Test the training generator by obtaining an image using the `next` generator function, and then using imshow to plot it.
# Print the corresponding label
from matplotlib.pyplot import imshow
image, label = next(training_generator)
image_unbatched = image[0,:,:,:]
imshow(image_unbatched)
print(label)

[[0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]

In [75]:

# Reset the generator by re-running the `get_generator` function.
train_generator = get_generator(training_features, training_labels)

Create a data augmention generator¶

In [76]:

from tensorflow.keras.preprocessing.image import ImageDataGenerator

In [77]:

# Create a function to convert an image to monochrome
def monochrome(x):
    def func_bw(a):
        average_colour = np.mean(a)
        return [average_colour, average_colour, average_colour]
    x = np.apply_along_axis(func_bw, -1, x)
    return x

In [78]:

# Create an ImageDataGenerator object
image_generator = ImageDataGenerator (
    preprocessing_function = monochrome,
    rotation_range = 180,
    rescale = (1 / 255.0)
)
image_generator.fit(training_features)

Check the documentation for the full list of image data augmentation options.

In [79]:

# Create an iterable generator using the `flow` function
image_generator_iterable = image_generator.flow (
    training_features, training_labels, batch_size = 1, shuffle = False
)

In [80]:

# Show a sample from the generator and compare with the original
image, label = next(image_generator_iterable)
image_orig, label_orig = next(train_generator)
figs, axes = plt.subplots(1,2)
axes[0].imshow(image[0,:,:,:])
axes[0].set_title('Transformed')
axes[1].imshow(image_orig[0,:,:,:])
axes[1].set_title('Original')
plt.show()

Flow from directory¶

In [81]:

! ls data/flowers-recognition-split/

train  val

In [82]:

# Inspect the directory structure
train_path = 'data/flowers-recognition-split/train'
val_path = 'data/flowers-recognition-split/val'

In [83]:

# Create an ImageDataGenerator object
datagenerator = ImageDataGenerator(rescale = (1 / 255.0))

In [84]:

classes = ['daisy', 'dandelion', 'rose', 'sunflower', 'tulip']

In [85]:

# Create a training data generator
train_generator = datagenerator.flow_from_directory (
    train_path, batch_size = 64, classes = classes, target_size = (16, 16)
)

Found 3027 images belonging to 5 classes.

In [86]:

# Create a validation data generator
val_generator = datagenerator.flow_from_directory (
    val_path, batch_size = 64, classes = classes, target_size = (16, 16)
)

Found 1296 images belonging to 5 classes.

In [87]:

# Get and display an image and label from the training generator
x = next(train_generator)
imshow(x[0][4])
print(x[1][4])

[0. 0. 0. 1. 0.]

In [88]:

# Reset the training generator
train_generator = datagenerator.flow_from_directory (
    train_path, batch_size = 64, classes = classes, target_size = (16, 16)
)

Found 3027 images belonging to 5 classes.

Create a model to train¶

In [89]:

# Build a CNN model
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Input, Flatten, Dense
model = tf.keras.Sequential()
model.add(Input((16, 16, 3)))
model.add(Conv2D(8, (8, 8), padding = 'same', activation = 'relu'))
model.add(MaxPooling2D((4, 4)))
model.add(Conv2D(8, (8, 8), padding = 'same', activation = 'relu'))
model.add(MaxPooling2D((2, 2)))
model.add(Conv2D(4, (4, 4), padding = 'same', activation = 'relu'))
model.add(Flatten())
model.add(Dense(16, activation = 'relu'))
model.add(Dense(8, activation = 'relu'))
model.add(Dense(5, activation = 'softmax'))

In [90]:

# Create an optimizer object
optimizer = tf.keras.optimizers.Adam(1e-3)

In [91]:

# Compile the model
model.compile(optimizer = optimizer, loss = 'categorical_crossentropy', metrics = ['accuracy'])

In [92]:

# Print the model summary
model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 16, 16, 8)         1544      
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 4, 4, 8)           0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 4, 4, 8)           4104      
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 2, 2, 8)           0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 2, 2, 4)           516       
_________________________________________________________________
flatten (Flatten)            (None, 16)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 16)                272       
_________________________________________________________________
dense_3 (Dense)              (None, 8)                 136       
_________________________________________________________________
dense_4 (Dense)              (None, 5)                 45        
=================================================================
Total params: 6,617
Trainable params: 6,617
Non-trainable params: 0
_________________________________________________________________

Train the model¶

In [93]:

# Calculate the training generator and test generator steps per epoch
train_steps_per_epoch = train_generator.n // train_generator.batch_size
val_steps = val_generator.n // val_generator.batch_size
print(train_steps_per_epoch, val_steps)

47 20

In [94]:

# Fit the model
model.fit_generator(train_generator, steps_per_epoch = train_steps_per_epoch, epochs = 5)

Epoch 1/5
47/47 [==============================] - 30s 634ms/step - loss: 1.5690 - accuracy: 0.3148
Epoch 2/5
47/47 [==============================] - 16s 340ms/step - loss: 1.4544 - accuracy: 0.3459
Epoch 3/5
47/47 [==============================] - 16s 332ms/step - loss: 1.3900 - accuracy: 0.3770
Epoch 4/5
47/47 [==============================] - 16s 330ms/step - loss: 1.3519 - accuracy: 0.3908
Epoch 5/5
47/47 [==============================] - 15s 329ms/step - loss: 1.3046 - accuracy: 0.4134

Out[94]:

<tensorflow.python.keras.callbacks.History at 0x7fad157a2b10>

Evaluate the model¶

In [95]:

# Evaluate the model
model.evaluate_generator(val_generator, steps = val_steps)

WARNING:tensorflow:From <ipython-input-95-f4aa522429b9>:2: Model.evaluate_generator (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
Please use Model.evaluate, which supports generators.

Out[95]:

[1.370197057723999, 0.38593751192092896]

Predict using the generator¶

In [96]:

# Predict labels with the model
predictions = model.predict_generator(val_generator, steps = 1)
print(np.round(predictions, 2))

[[0.08 0.3  0.13 0.28 0.21]
 [0.33 0.48 0.06 0.05 0.07]
 [0.51 0.26 0.12 0.03 0.08]
 [0.73 0.19 0.05 0.   0.02]
 [0.21 0.52 0.07 0.09 0.1 ]
 [0.34 0.57 0.03 0.02 0.04]
 [0.29 0.54 0.05 0.05 0.07]
 [0.5  0.31 0.1  0.02 0.08]
 [0.11 0.41 0.11 0.19 0.18]
 [0.38 0.42 0.08 0.04 0.08]
 [0.39 0.54 0.03 0.01 0.03]
 [0.13 0.03 0.52 0.03 0.29]
 [0.28 0.44 0.09 0.07 0.11]
 [0.18 0.36 0.15 0.12 0.2 ]
 [0.27 0.51 0.07 0.05 0.09]
 [0.19 0.36 0.15 0.11 0.2 ]
 [0.43 0.52 0.02 0.01 0.02]
 [0.   0.   0.56 0.03 0.41]
 [0.13 0.68 0.03 0.1  0.07]
 [0.09 0.4  0.1  0.24 0.17]
 [0.04 0.01 0.54 0.05 0.36]
 [0.13 0.22 0.22 0.16 0.27]
 [0.39 0.25 0.18 0.05 0.13]
 [0.01 0.   0.5  0.06 0.44]
 [0.1  0.26 0.16 0.23 0.24]
 [0.14 0.12 0.33 0.1  0.3 ]
 [0.02 0.01 0.54 0.04 0.39]
 [0.55 0.25 0.11 0.02 0.07]
 [0.16 0.65 0.04 0.07 0.08]
 [0.42 0.3  0.13 0.03 0.11]
 [0.08 0.07 0.41 0.07 0.37]
 [0.14 0.59 0.06 0.1  0.11]
 [0.39 0.21 0.21 0.04 0.15]
 [0.16 0.03 0.54 0.02 0.25]
 [0.42 0.39 0.09 0.03 0.07]
 [0.27 0.58 0.04 0.04 0.06]
 [0.13 0.56 0.06 0.15 0.11]
 [0.65 0.31 0.03 0.   0.02]
 [0.1  0.04 0.49 0.04 0.33]
 [0.03 0.02 0.42 0.11 0.42]
 [0.2  0.06 0.45 0.03 0.26]
 [0.58 0.34 0.04 0.01 0.03]
 [0.23 0.11 0.36 0.05 0.24]
 [0.16 0.26 0.21 0.11 0.25]
 [0.08 0.09 0.32 0.15 0.36]
 [0.65 0.29 0.03 0.   0.02]
 [0.31 0.1  0.35 0.03 0.21]
 [0.22 0.61 0.05 0.05 0.07]
 [0.15 0.21 0.24 0.12 0.27]
 [0.19 0.67 0.03 0.06 0.06]
 [0.41 0.18 0.23 0.03 0.15]
 [0.32 0.65 0.01 0.01 0.02]
 [0.45 0.33 0.11 0.02 0.09]
 [0.35 0.42 0.08 0.05 0.09]
 [0.64 0.23 0.08 0.   0.04]
 [0.07 0.08 0.35 0.13 0.37]
 [0.35 0.56 0.03 0.02 0.04]
 [0.06 0.04 0.42 0.1  0.39]
 [0.07 0.01 0.59 0.02 0.31]
 [0.12 0.52 0.07 0.16 0.13]
 [0.53 0.23 0.14 0.02 0.09]
 [0.05 0.12 0.25 0.21 0.37]
 [0.65 0.18 0.1  0.01 0.05]
 [0.02 0.01 0.58 0.03 0.36]]

---

The Dataset Class¶

In [97]:

import matplotlib.pyplot as plt
import numpy as np
import os

Create a simple dataset¶

In [98]:

x = np.zeros((100, 10, 2, 2))

In [99]:

# Create a dataset from the tensor x
dataset1 = tf.data.Dataset.from_tensor_slices(x)

In [100]:

# Inspect the Dataset object
print(dataset1)
print(dataset1.element_spec)

<TensorSliceDataset shapes: (10, 2, 2), types: tf.float64>
TensorSpec(shape=(10, 2, 2), dtype=tf.float64, name=None)

In [101]:

x2 = [np.zeros((10, 2, 2)), np.zeros((5, 2, 2))]

In [102]:

# Try creating a dataset from the tensor x2
dataset2 = tf.data.Dataset.from_tensor_slices(x2)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-102-9b60f1d069ed> in <module>
      1 # Try creating a dataset from the tensor x2
----> 2 dataset2 = tf.data.Dataset.from_tensor_slices(x2)

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py in from_tensor_slices(tensors)
    638       Dataset: A `Dataset`.
    639     """
--> 640     return TensorSliceDataset(tensors)
    641 
    642   class _GeneratorState(object):

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py in __init__(self, element)
   2856   def __init__(self, element):
   2857     """See `Dataset.from_tensor_slices()` for details."""
-> 2858     element = structure.normalize_element(element)
   2859     batched_spec = structure.type_spec_from_value(element)
   2860     self._tensors = structure.to_batched_tensor_list(batched_spec, element)

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/data/util/structure.py in normalize_element(element)
    113         else:
    114           normalized_components.append(
--> 115               ops.convert_to_tensor(t, name="component_%d" % i))
    116   return nest.pack_sequence_as(element, normalized_components)
    117 

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, dtype_hint, ctx, accepted_result_types)
   1339 
   1340     if ret is None:
-> 1341       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
   1342 
   1343     if ret is NotImplemented:

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py in _constant_tensor_conversion_function(v, dtype, name, as_ref)
    319                                          as_ref=False):
    320   _ = as_ref
--> 321   return constant(v, dtype=dtype, name=name)
    322 
    323 

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name)
    260   """
    261   return _constant_impl(value, dtype, shape, name, verify_shape=False,
--> 262                         allow_broadcast=True)
    263 
    264 

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py in _constant_impl(value, dtype, shape, name, verify_shape, allow_broadcast)
    268   ctx = context.context()
    269   if ctx.executing_eagerly():
--> 270     t = convert_to_eager_tensor(value, ctx, dtype)
    271     if shape is None:
    272       return t

~/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py in convert_to_eager_tensor(value, ctx, dtype)
     94       dtype = dtypes.as_dtype(dtype).as_datatype_enum
     95   ctx.ensure_initialized()
---> 96   return ops.EagerTensor(value, ctx.device_name, dtype)
     97 
     98 

ValueError: Can't convert non-rectangular Python sequence to Tensor.

In [103]:

x2 = [np.zeros((10, 1)), np.zeros((10, 1)), np.zeros((10, 1))]

In [104]:

# Create another dataset from the new x2 and inspect the Dataset object
dataset2 = tf.data.Dataset.from_tensor_slices(x2)

In [105]:

# Print the element_spec
print(dataset2.element_spec)

TensorSpec(shape=(10, 1), dtype=tf.float64, name=None)

Create a zipped dataset¶

In [106]:

# Combine the two datasets into one larger dataset
dataset_zipped = tf.data.Dataset.zip((dataset1, dataset2))

In [107]:

# Print the element_spec
print(dataset_zipped.element_spec)

(TensorSpec(shape=(10, 2, 2), dtype=tf.float64, name=None), TensorSpec(shape=(10, 1), dtype=tf.float64, name=None))

In [108]:

# Define a function to find the number of batches in a dataset
def get_batches(dataset):
    iter_dataset = iter(dataset)
    i = 0
    try:
        while next(iter_dataset): i = i + 1
    except:
        return i

In [109]:

# Find the number of batches in the zipped Dataset
get_batches(dataset_zipped)

Out[109]:

3

Create a dataset from numpy arrays¶

In [110]:

# Load the MNIST dataset
(train_features, train_labels), (test_features, test_labels) = tf.keras.datasets.mnist.load_data()
print(type(train_features), type(train_labels))

<class 'numpy.ndarray'> <class 'numpy.ndarray'>

In [111]:

# Create a Dataset from the MNIST data
mnist_dataset = tf.data.Dataset.from_tensor_slices((train_features, train_labels))

In [112]:

# Inspect the Dataset object
print(mnist_dataset.element_spec)

(TensorSpec(shape=(28, 28), dtype=tf.uint8, name=None), TensorSpec(shape=(), dtype=tf.uint8, name=None))

In [113]:

# Inspect the length of an element using the take method
element = next(iter(mnist_dataset.take(1)))
print(len(element))

2

In [114]:

# Examine the shapes of the data
print(element[0].shape)
print(element[1].shape)

(28, 28)
()

Create a dataset from text data¶

In [115]:

# Print the list of text files
text_files = sorted([f.path for f in os.scandir('data/shakespeare')])
print(text_files)

['data/shakespeare/tempest.1.1.txt', 'data/shakespeare/tempest.1.2.txt', 'data/shakespeare/tempest.2.1.txt', 'data/shakespeare/tempest.2.2.txt', 'data/shakespeare/tempest.3.1.txt', 'data/shakespeare/tempest.3.2.txt', 'data/shakespeare/tempest.3.3.txt', 'data/shakespeare/tempest.4.1.txt', 'data/shakespeare/tempest.5.1.txt']

In [116]:

# Load the first file using python and print the first 5 lines.
with open(text_files[0], 'r') as fil:
    contents = [fil.readline() for i in range(5)]
    for line in contents: print(line)

SCENE I. On a ship at sea: a tempestuous noise

of thunder and lightning heard.

Enter a Master and a Boatswain



Master

In [117]:

# Load the lines from the files into a dataset using TextLineDataset
shakespeare_dataset = tf.data.TextLineDataset(text_files)

In [118]:

# Use the take method to get and print the first 5 lines of the dataset
first_5_lines_dataset = iter(shakespeare_dataset.take(5))
lines = [line for line in first_5_lines_dataset]
for line in lines:
    print(line)

tf.Tensor(b'SCENE I. On a ship at sea: a tempestuous noise', shape=(), dtype=string)
tf.Tensor(b'of thunder and lightning heard.', shape=(), dtype=string)
tf.Tensor(b'Enter a Master and a Boatswain', shape=(), dtype=string)
tf.Tensor(b'', shape=(), dtype=string)
tf.Tensor(b'Master', shape=(), dtype=string)

In [119]:

# Compute the number of lines in the first file
lines = []
with open(text_files[0], 'r') as fil:
    line = fil.readline()
    while line:
        lines.append(line)
        line = fil.readline()
    print(len(lines))

121

In [120]:

# Compute the number of lines in the shakespeare dataset we created
shakespeare_dataset_iterator = iter(shakespeare_dataset)
lines = [line for line in shakespeare_dataset_iterator]
print(len(lines))

3134

Interleave lines from the text data files¶

In [121]:

# Create a dataset of the text file strings
text_files_dataset = tf.data.Dataset.from_tensor_slices(text_files)
files = [file for file in text_files_dataset]
for file in files:
    print(file)

tf.Tensor(b'data/shakespeare/tempest.1.1.txt', shape=(), dtype=string)
tf.Tensor(b'data/shakespeare/tempest.1.2.txt', shape=(), dtype=string)
tf.Tensor(b'data/shakespeare/tempest.2.1.txt', shape=(), dtype=string)
tf.Tensor(b'data/shakespeare/tempest.2.2.txt', shape=(), dtype=string)
tf.Tensor(b'data/shakespeare/tempest.3.1.txt', shape=(), dtype=string)
tf.Tensor(b'data/shakespeare/tempest.3.2.txt', shape=(), dtype=string)
tf.Tensor(b'data/shakespeare/tempest.3.3.txt', shape=(), dtype=string)
tf.Tensor(b'data/shakespeare/tempest.4.1.txt', shape=(), dtype=string)
tf.Tensor(b'data/shakespeare/tempest.5.1.txt', shape=(), dtype=string)

In [122]:

# Interleave the lines from the text files
interleaved_shakespeare_dataset = text_files_dataset.interleave(tf.data.TextLineDataset, cycle_length = 9)
print(interleaved_shakespeare_dataset.element_spec)

TensorSpec(shape=(), dtype=tf.string, name=None)

In [123]:

# Print the first 10 elements of the interleaved dataset
lines = [line for line in iter(interleaved_shakespeare_dataset.take(10))]
for line in lines:
    print(line)

tf.Tensor(b'SCENE I. On a ship at sea: a tempestuous noise', shape=(), dtype=string)
tf.Tensor(b"SCENE II. The island. Before PROSPERO'S cell.", shape=(), dtype=string)
tf.Tensor(b'SCENE I. Another part of the island.', shape=(), dtype=string)
tf.Tensor(b'SCENE II. Another part of the island.', shape=(), dtype=string)
tf.Tensor(b"SCENE I. Before PROSPERO'S Cell.", shape=(), dtype=string)
tf.Tensor(b'SCENE II. Another part of the island.', shape=(), dtype=string)
tf.Tensor(b'SCENE III. Another part of the island.', shape=(), dtype=string)
tf.Tensor(b"SCENE I. Before PROSPERO'S cell.", shape=(), dtype=string)
tf.Tensor(b"SCENE I. Before PROSPERO'S cell.", shape=(), dtype=string)
tf.Tensor(b'of thunder and lightning heard.', shape=(), dtype=string)

---

Training with Datasets¶

In [124]:

import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd

Load the UCI Bank Marketing Dataset¶

In [125]:

# Load the CSV file into a pandas DataFrame
bank_dataframe = pd.read_csv('data/bank/bank-full.csv', delimiter=';')

In [126]:

# Show the head of the DataFrame
bank_dataframe.head()

Out[126]:

	age	job	marital	education	default	balance	housing	loan	contact	day	month	duration	campaign	pdays	previous	poutcome	y
0	58	management	married	tertiary	no	2143	yes	no	unknown	5	may	261	1	-1	0	unknown	no
1	44	technician	single	secondary	no	29	yes	no	unknown	5	may	151	1	-1	0	unknown	no
2	33	entrepreneur	married	secondary	no	2	yes	yes	unknown	5	may	76	1	-1	0	unknown	no
3	47	blue-collar	married	unknown	no	1506	yes	no	unknown	5	may	92	1	-1	0	unknown	no
4	33	unknown	single	unknown	no	1	no	no	unknown	5	may	198	1	-1	0	unknown	no

In [127]:

# Print the shape of the DataFrame
print(bank_dataframe.shape)

(45211, 17)

In [128]:

# Select features from the DataFrame
features = [
    'age', 'job', 'marital', 'education', 'default', 'balance', 'housing',
    'loan', 'contact', 'campaign', 'pdays', 'poutcome'
]
labels = ['y']
bank_dataframe = bank_dataframe.filter(features + labels)

In [129]:

# Show the head of the DataFrame
bank_dataframe.head()

Out[129]:

	age	job	marital	education	default	balance	housing	loan	contact	campaign	pdays	poutcome	y
0	58	management	married	tertiary	no	2143	yes	no	unknown	1	-1	unknown	no
1	44	technician	single	secondary	no	29	yes	no	unknown	1	-1	unknown	no
2	33	entrepreneur	married	secondary	no	2	yes	yes	unknown	1	-1	unknown	no
3	47	blue-collar	married	unknown	no	1506	yes	no	unknown	1	-1	unknown	no
4	33	unknown	single	unknown	no	1	no	no	unknown	1	-1	unknown	no

Preprocess the data¶

In [130]:

# Convert the categorical features in the DataFrame to one-hot encodings

from sklearn.preprocessing import LabelBinarizer

encoder = LabelBinarizer()
categorical_features = ['default', 'housing', 'job', 'loan', 'education', 'contact', 'poutcome']

for feature in categorical_features:
    bank_dataframe[feature] = tuple(encoder.fit_transform(bank_dataframe[feature]))

In [131]:

# Show the head of the DataFrame
bank_dataframe.head()

Out[131]:

	age	job	marital	education	default	balance	housing	loan	contact	campaign	pdays	poutcome	y
0	58	(0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0)	married	(0, 0, 1, 0)	(0,)	2143	(1,)	(0,)	(0, 0, 1)	1	-1	(0, 0, 0, 1)	no
1	44	(0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0)	single	(0, 1, 0, 0)	(0,)	29	(1,)	(0,)	(0, 0, 1)	1	-1	(0, 0, 0, 1)	no
2	33	(0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0)	married	(0, 1, 0, 0)	(0,)	2	(1,)	(1,)	(0, 0, 1)	1	-1	(0, 0, 0, 1)	no
3	47	(0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)	married	(0, 0, 0, 1)	(0,)	1506	(1,)	(0,)	(0, 0, 1)	1	-1	(0, 0, 0, 1)	no
4	33	(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1)	single	(0, 0, 0, 1)	(0,)	1	(0,)	(0,)	(0, 0, 1)	1	-1	(0, 0, 0, 1)	no

In [132]:

# Shuffle the DataFrame
bank_dataframe = bank_dataframe.sample(frac = 1).reset_index(drop = True)

Create the Dataset object¶

In [133]:

# Convert the DataFrame to a Dataset
bank_dataset = tf.data.Dataset.from_tensor_slices(bank_dataframe.to_dict('list')) 

In [134]:

# Inspect the Dataset object
bank_dataset.element_spec

Out[134]:

{'age': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'job': TensorSpec(shape=(12,), dtype=tf.int32, name=None),
 'marital': TensorSpec(shape=(), dtype=tf.string, name=None),
 'education': TensorSpec(shape=(4,), dtype=tf.int32, name=None),
 'default': TensorSpec(shape=(1,), dtype=tf.int32, name=None),
 'balance': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'housing': TensorSpec(shape=(1,), dtype=tf.int32, name=None),
 'loan': TensorSpec(shape=(1,), dtype=tf.int32, name=None),
 'contact': TensorSpec(shape=(3,), dtype=tf.int32, name=None),
 'campaign': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'pdays': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'poutcome': TensorSpec(shape=(4,), dtype=tf.int32, name=None),
 'y': TensorSpec(shape=(), dtype=tf.string, name=None)}

Filter the Dataset¶

In [135]:

# First check that there are records in the dataset for non-married individuals

def check_divorced():
    bank_dataset_iterable = iter(bank_dataset)
    for x in bank_dataset_iterable:
        if x['marital'] != 'divorced':
            print('Found a person with marital status: {}'.format(x['marital']))
            return
    print('No non-divorced people were found!')

check_divorced()

Found a person with marital status: b'single'

In [136]:

# Filter the Dataset to retain only entries with a 'divorced' marital status
bank_dataset = bank_dataset.filter(lambda x : tf.equal(x['marital'], tf.constant([b'divorced']))[0] )

In [137]:

# Check the records in the dataset again
check_divorced()

No non-divorced people were found!

Map a function over the dataset¶

In [138]:

# Convert the label ('y') to an integer instead of 'yes' or 'no'
def map_label(x):
    x['y'] = 0 if (x['y'] == tf.constant([b'no'], dtype = tf.string)) else 1
    return x
bank_dataset = bank_dataset.map(map_label)

In [139]:

# Inspect the Dataset object
bank_dataset.element_spec

Out[139]:

{'age': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'job': TensorSpec(shape=(12,), dtype=tf.int32, name=None),
 'marital': TensorSpec(shape=(), dtype=tf.string, name=None),
 'education': TensorSpec(shape=(4,), dtype=tf.int32, name=None),
 'default': TensorSpec(shape=(1,), dtype=tf.int32, name=None),
 'balance': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'housing': TensorSpec(shape=(1,), dtype=tf.int32, name=None),
 'loan': TensorSpec(shape=(1,), dtype=tf.int32, name=None),
 'contact': TensorSpec(shape=(3,), dtype=tf.int32, name=None),
 'campaign': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'pdays': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'poutcome': TensorSpec(shape=(4,), dtype=tf.int32, name=None),
 'y': TensorSpec(shape=(), dtype=tf.int32, name=None)}

In [140]:

# Remove the 'marital' column
bank_dataset = bank_dataset.map(lambda x: {key: val for key, val in x.items() if key != 'marital'})

In [141]:

# Inspect the Dataset object
bank_dataset.element_spec

Out[141]:

{'age': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'job': TensorSpec(shape=(12,), dtype=tf.int32, name=None),
 'education': TensorSpec(shape=(4,), dtype=tf.int32, name=None),
 'default': TensorSpec(shape=(1,), dtype=tf.int32, name=None),
 'balance': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'housing': TensorSpec(shape=(1,), dtype=tf.int32, name=None),
 'loan': TensorSpec(shape=(1,), dtype=tf.int32, name=None),
 'contact': TensorSpec(shape=(3,), dtype=tf.int32, name=None),
 'campaign': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'pdays': TensorSpec(shape=(), dtype=tf.int32, name=None),
 'poutcome': TensorSpec(shape=(4,), dtype=tf.int32, name=None),
 'y': TensorSpec(shape=(), dtype=tf.int32, name=None)}

Create input and output data tuples¶

In [142]:

# Create an input and output tuple for the dataset
def map_feature_label(x):
    features = [[x['age']], [x['balance']], [x['campaign']], x['contact'], x['default'],
                x['education'], x['housing'], x['job'], x['loan'], [x['pdays']], x['poutcome']]
    return (tf.concat(features, axis = 0), x['y'])

In [143]:

# Map this function over the dataset
bank_dataset = bank_dataset.map(map_feature_label)

In [144]:

# Inspect the Dataset object
bank_dataset.element_spec

Out[144]:

(TensorSpec(shape=(30,), dtype=tf.int32, name=None),
 TensorSpec(shape=(), dtype=tf.int32, name=None))

Split into a training and a validation set¶

In [145]:

# Determine the length of the Dataset
dataset_length = 0
for _ in bank_dataset:
    dataset_length += 1
print(dataset_length)

5207

In [146]:

# Make training and validation sets from the dataset
training_elements = int(dataset_length * 0.7)
train_dataset = bank_dataset.take(training_elements)
validation_dataset = bank_dataset.skip(training_elements)

Build a classification model¶

Now let's build a model to classify the features.

In [147]:

# Build a classifier model
from tensorflow.keras.layers import Dense, Input, Concatenate, BatchNormalization
from tensorflow.keras import Sequential

model = Sequential()
model.add(Input(shape=(30,)))
model.add(BatchNormalization(momentum = 0.8))
model.add(Dense(400, activation = 'relu'))
model.add(BatchNormalization(momentum = 0.8))
model.add(Dense(400, activation = 'relu'))
model.add(BatchNormalization(momentum = 0.8))
model.add(Dense(1, activation = 'sigmoid'))

In [148]:

# Compile the model
optimizer = tf.keras.optimizers.Adam(1e-4)
model.compile(optimizer = optimizer, loss = 'binary_crossentropy', metrics = ['accuracy'])

In [149]:

# Show the model summary
model.summary()

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
batch_normalization_2 (Batch (None, 30)                120       
_________________________________________________________________
dense_5 (Dense)              (None, 400)               12400     
_________________________________________________________________
batch_normalization_3 (Batch (None, 400)               1600      
_________________________________________________________________
dense_6 (Dense)              (None, 400)               160400    
_________________________________________________________________
batch_normalization_4 (Batch (None, 400)               1600      
_________________________________________________________________
dense_7 (Dense)              (None, 1)                 401       
=================================================================
Total params: 176,521
Trainable params: 174,861
Non-trainable params: 1,660
_________________________________________________________________

Train the model¶

In [150]:

# Create batched training and validation datasets
train_dataset = train_dataset.batch(20, drop_remainder = True)
validation_dataset = validation_dataset.batch(100)

In [151]:

# Shuffle the training data
train_dataset = train_dataset.shuffle(1000)

In [152]:

# Fit the model
history = model.fit(train_dataset, validation_data = validation_dataset, epochs = 5)

Epoch 1/5
182/182 [==============================] - 8s 44ms/step - loss: 0.6631 - accuracy: 0.6431 - val_loss: 0.6044 - val_accuracy: 0.7658
Epoch 2/5
182/182 [==============================] - 8s 44ms/step - loss: 0.4845 - accuracy: 0.8212 - val_loss: 0.5670 - val_accuracy: 0.8042
Epoch 3/5
182/182 [==============================] - 8s 44ms/step - loss: 0.3791 - accuracy: 0.8893 - val_loss: 0.4822 - val_accuracy: 0.8241
Epoch 4/5
182/182 [==============================] - 8s 45ms/step - loss: 0.3103 - accuracy: 0.9102 - val_loss: 0.4032 - val_accuracy: 0.8727
Epoch 5/5
182/182 [==============================] - 8s 44ms/step - loss: 0.2611 - accuracy: 0.9255 - val_loss: 0.4690 - val_accuracy: 0.8580

In [153]:

# Plot the training and validation accuracy
plt.plot(history.epoch, history.history['accuracy'], label = 'training')
plt.plot(history.epoch, history.history['val_accuracy'], label = 'validation')
plt.legend()
plt.xlabel('Epoch')
plt.ylabel('Accuracy')

Out[153]:

Text(0, 0.5, 'Accuracy')

Tokenising Text Data

In this notebook, you will learn how to tokenise text data using tf.keras.preprocessing.text.Tokenizer.

In [1]:

import tensorflow as tf
tf.__version__

Out[1]:

'2.2.0'

You have now downloaded and experimented with the IMDb dataset of labelled movie reviews. You will have noticed that the words have been mapped to integers. Converting a sequence of words to a sequence of numbers is called tokenisation. The numbers themselves are called tokens. Tokenisation is handy because it allows numerical operations to be applied to text data.

The IMDb reviews were tokenised by mapping each word to a positive integer that indicated its frequency rank. Tokenisation could also have been applied at the level of characters rather than words.

The text dataset¶

The text we will work with in this notebook is Three Men in a Boat by Jerome K. Jerome, a comical short story about the perils of going outside.

In [2]:

# Load the data
with open('data/ThreeMenInABoat.txt', 'r', encoding = 'utf-8') as file:
    text_string = file.read().replace('\n', ' ')

In [3]:

# Perform some simple preprocessing, replacing dashes with empty spaces
text_string = text_string.replace('—', '')

In [4]:

# View an excerpt of the data
text_string[0:2001]

Out[4]:

'CHAPTER I.   Three invalids.Sufferings of George and Harris.A victim to one hundred and seven fatal maladies.Useful prescriptions.Cure for liver complaint in children.We agree that we are overworked, and need rest.A week on the rolling deep?George suggests the River.Montmorency lodges an objection.Original motion carried by majority of three to one.  There were four of usGeorge, and William Samuel Harris, and myself, and Montmorency.  We were sitting in my room, smoking, and talking about how bad we werebad from a medical point of view I mean, of course.  We were all feeling seedy, and we were getting quite nervous about it. Harris said he felt such extraordinary fits of giddiness come over him at times, that he hardly knew what he was doing; and then George said that _he_ had fits of giddiness too, and hardly knew what _he_ was doing. With me, it was my liver that was out of order.  I knew it was my liver that was out of order, because I had just been reading a patent liver-pill circular, in which were detailed the various symptoms by which a man could tell when his liver was out of order.  I had them all.  It is a most extraordinary thing, but I never read a patent medicine advertisement without being impelled to the conclusion that I am suffering from the particular disease therein dealt with in its most virulent form.  The diagnosis seems in every case to correspond exactly with all the sensations that I have ever felt.  [Picture: Man reading book] I remember going to the British Museum one day to read up the treatment for some slight ailment of which I had a touchhay fever, I fancy it was.  I got down the book, and read all I came to read; and then, in an unthinking moment, I idly turned the leaves, and began to indolently study diseases, generally.  I forget which was the first distemper I plunged intosome fearful, devastating scourge, I knowand, before I had glanced half down the list of “premonitory symptoms,” it was borne in upon me that I had fairly got it.'

In [5]:

# Split the text into sentences.
sentence_strings = text_string.split('.')

In [6]:

# View a sample of the dataset
sentence_strings[20:30]

Out[6]:

['  I got down the book, and read all I came to read; and then, in an unthinking moment, I idly turned the leaves, and began to indolently study diseases, generally',
 '  I forget which was the first distemper I plunged intosome fearful, devastating scourge, I knowand, before I had glanced half down the list of “premonitory symptoms,” it was borne in upon me that I had fairly got it',
 '  I sat for awhile, frozen with horror; and then, in the listlessness of despair, I again turned over the pages',
 '  I came to typhoid feverread the symptomsdiscovered that I had typhoid fever, must have had it for months without knowing itwondered what else I had got; turned up St',
 ' Vitus’s Dancefound, as I expected, that I had that too,began to get interested in my case, and determined to sift it to the bottom, and so started alphabeticallyread up ague, and learnt that I was sickening for it, and that the acute stage would commence in about another fortnight',
 '  Bright’s disease, I was relieved to find, I had only in a modified form, and, so far as that was concerned, I might live for years',
 '  Cholera I had, with severe complications; and diphtheria I seemed to have been born with',
 '  I plodded conscientiously through the twenty-six letters, and the only malady I could conclude I had not got was housemaid’s knee',
 '  I felt rather hurt about this at first; it seemed somehow to be a sort of slight',
 '  Why hadn’t I got housemaid’s knee?  Why this invidious reservation?  After a while, however, less grasping feelings prevailed']

Create a Tokenizer object¶

The Tokenizer object allows you to easily tokenise words or characters from a text document. It has several options to allow you to adjust the tokenisation process. Documentation is available for the Tokenizer here.

In [7]:

# Define any additional characters that we want to filter out (ignore) from the text
additional_filters = '—’‘“”'

The Tokenizer has a filters keyword argument, that determines which characters will be filtered out from the text. The cell below shows the default characters that are filtered, to which we are adding our additional filters.

In [8]:

# Create a Tokenizer object
from tensorflow.keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer (
    num_words = None,
    filters = '!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n' + additional_filters,
    lower = True,
    split = ' ',
    char_level = False,
    oov_token = '<UNK>',
    document_count = 0
)

In all, the Tokenizer has the following keyword arguments:

num_words: int. the maximum number of words to keep, based on word frequency. Only the most common num_words-1 words will be kept. If set to None, all words are kept.

filters: str. Each element is a character that will be filtered from the texts. Defaults to all punctuation (inc. tabs and line breaks), except '.

lower: bool. Whether to convert the texts to lowercase. Defaults to True.

split: str. Separator for word splitting. Defaults to ' '.

char_level: bool. if True, every character will be treated as a token. Defaults to False.

oov_token: if given, it will be added to word_index and used to replace out-of-vocabulary words during sequence_to_text calls. Defaults to None.

Fit the Tokenizer to the text¶

We can now tokenize our text using the fit_on_texts method. This method takes a list of strings to tokenize, as we have prepared with sentence_strings.

In [9]:

# Build the Tokenizer vocabulary
tokenizer.fit_on_texts(sentence_strings)

The fit_on_texts method could also take a list of lists of strings, and in this case it would recognise each element of each sublist as an individual token.

Get the Tokenizer configuration¶

Now that the Tokenizer has ingested the data, we can see what it has extracted from the text by viewing its configuration.

In [10]:

# Get the tokenizer config as a python dict
tokenizer_config = tokenizer.get_config()
tokenizer_config.keys()

Out[10]:

dict_keys(['num_words', 'filters', 'lower', 'split', 'char_level', 'oov_token', 'document_count', 'word_counts', 'word_docs', 'index_docs', 'index_word', 'word_index'])

In [11]:

# View the word_counts entry
tokenizer_config['word_counts']

Out[11]:

'{"chapter": 19, "i": 1195, "three": 79, "invalids": 1, "sufferings": 2, "of": 1487, "george": 306, "and": 3375, "harris": 314, "a": 1696, "victim": 3, "to": 1785, "one": 241, "hundred": 19, "seven": 15, "fatal": 1, "maladies": 2, "useful": 2, "prescriptions": 1, "cure": 1, "for": 525, "liver": 8, "complaint": 2, "in": 976, "children": 13, "we": 866, "agree": 2, "that": 944, "are": 181, "overworked": 1, "need": 7, "rest": 14, "week": 19, "on": 501, "the": 3603, "rolling": 1, "deep": 18, "suggests": 2, "river": 165, "montmorency": 61, "lodges": 1, "an": 186, "objection": 4, "original": 5, "motion": 3, "carried": 7, "by": 209, "majority": 3, "there": 326, "were": 238, "four": 39, "usgeorge": 1, "william": 4, "samuel": 1, "myself": 56, "sitting": 26, "my": 192, "room": 43, "smoking": 6, "talking": 10, "about": 223, "how": 92, "bad": 15, "werebad": 1, "from": 218, "medical": 6, "point": 18, "view": 16, "mean": 14, "course": 40, "all": 353, "feeling": 8, "seedy": 1, "getting": 29, "quite": 72, "nervous": 14, "it": 1405, "said": 375, "he": 917, "felt": 38, "such": 56, "extraordinary": 8, "fits": 4, "giddiness": 2, "come": 110, "over": 146, "him": 330, "at": 514, "times": 22, "hardly": 11, "knew": 41, "what": 204, "was": 777, "doing": 31, "then": 272, "had": 585, "too": 67, "with": 490, "me": 186, "out": 305, "order": 8, "because": 36, "just": 81, "been": 190, "reading": 26, "patent": 2, "pill": 2, "circular": 2, "which": 101, "detailed": 1, "various": 7, "symptoms": 3, "man": 164, "could": 153, "tell": 43, "when": 267, "his": 382, "them": 260, "is": 444, "most": 42, "thing": 91, "but": 357, "never": 113, "read": 11, "medicine": 1, "advertisement": 2, "without": 51, "being": 64, "impelled": 1, "conclusion": 5, "am": 31, "suffering": 6, "particular": 4, "disease": 5, "therein": 3, "dealt": 1, "its": 91, "virulent": 1, "form": 3, "diagnosis": 1, "seems": 19, "every": 47, "case": 19, "correspond": 1, "exactly": 13, "sensations": 1, "have": 338, "ever": 65, "picture": 82, "book": 6, "remember": 18, "going": 76, "british": 3, "museum": 1, "day": 95, "up": 482, "treatment": 1, "some": 107, "slight": 3, "ailment": 1, "touchhay": 1, "fever": 7, "fancy": 18, "got": 163, "down": 194, "came": 94, "unthinking": 1, "moment": 35, "idly": 1, "turned": 31, "leaves": 6, "began": 34, "indolently": 1, "study": 6, "diseases": 4, "generally": 11, "forget": 13, "first": 80, "distemper": 1, "plunged": 2, "intosome": 1, "fearful": 6, "devastating": 1, "scourge": 1, "knowand": 1, "before": 91, "glanced": 1, "half": 80, "list": 6, "premonitory": 1, "borne": 3, "upon": 98, "fairly": 4, "sat": 46, "awhile": 2, "frozen": 1, "horror": 1, "listlessness": 1, "despair": 1, "again": 67, "pages": 2, "typhoid": 3, "feverread": 1, "symptomsdiscovered": 1, "must": 56, "months": 9, "knowing": 8, "itwondered": 1, "else": 21, "st": 9, "vitus": 1, "s": 342, "dancefound": 1, "as": 386, "expected": 9, "get": 179, "interested": 4, "determined": 13, "sift": 1, "bottom": 17, "so": 275, "started": 25, "alphabeticallyread": 1, "ague": 1, "learnt": 4, "sickening": 1, "acute": 1, "stage": 10, "would": 357, "commence": 3, "another": 63, "fortnight": 6, "bright": 14, "relieved": 1, "find": 31, "only": 92, "modified": 2, "far": 27, "concerned": 4, "might": 44, "live": 14, "years": 33, "cholera": 2, "severe": 5, "complications": 1, "diphtheria": 1, "seemed": 97, "born": 8, "plodded": 1, "conscientiously": 2, "through": 52, "twenty": 17, "six": 26, "letters": 2, "malady": 2, "conclude": 1, "not": 386, "housemaid": 5, "knee": 5, "rather": 37, "hurt": 4, "this": 185, "somehow": 13, "be": 324, "sort": 38, "why": 69, "hadn": 11, "t": 282, "invidious": 1, "reservation": 1, "after": 108, "while": 72, "however": 37, "less": 13, "grasping": 4, "feelings": 5, "prevailed": 2, "reflected": 2, "other": 128, "known": 14, "pharmacology": 1, "grew": 8, "selfish": 2, "do": 147, "gout": 1, "malignant": 1, "appear": 5, "seized": 5, "aware": 1, "zymosis": 2, "evidently": 18, "boyhood": 1, "no": 124, "more": 134, "concluded": 3, "nothing": 31, "matter": 37, "pondered": 2, "thought": 119, "interesting": 10, "acquisition": 1, "should": 94, "class": 7, "students": 3, "walk": 22, "hospitals": 1, "if": 189, "they": 459, "hospital": 1, "round": 140, "take": 88, "their": 115, "diploma": 1, "wondered": 7, "long": 48, "tried": 35, "examine": 3, "pulse": 4, "feel": 41, "any": 95, "sudden": 2, "start": 17, "off": 85, "pulled": 21, "watch": 21, "timed": 1, "made": 69, "forty": 11, "minute": 10, "heart": 16, "stopped": 16, "beating": 3, "since": 15, "induced": 1, "opinion": 9, "time": 138, "cannot": 15, "account": 8, "patted": 1, "front": 12, "call": 20, "waist": 1, "head": 50, "went": 101, "bit": 52, "each": 61, "side": 55, "little": 113, "way": 102, "back": 115, "or": 183, "hear": 24, "anything": 39, "look": 52, "tongue": 3, "stuck": 10, "go": 131, "shut": 7, "eye": 12, "see": 97, "tip": 2, "gain": 4, "certain": 9, "than": 78, "scarlet": 2, "walking": 11, "stick": 7, "walked": 6, "into": 162, "happy": 12, "healthy": 2, "crawled": 1, "decrepit": 1, "wreck": 3, "old": 125, "chum": 2, "mine": 14, "feels": 8, "looks": 9, "talks": 1, "weather": 17, "m": 35, "ill": 17, "good": 104, "turn": 17, "now": 101, "doctor": 3, "wants": 6, "practice": 11, "shall": 18, "will": 87, "seventeen": 1, "your": 96, "ordinary": 5, "commonplace": 6, "patients": 1, "two": 124, "straight": 16, "saw": 39, "well": 81, "you": 643, "dear": 8, "boy": 56, "telling": 10, "life": 48, "brief": 4, "pass": 24, "away": 85, "finished": 14, "fact": 16, "remains": 4, "everything": 27, "told": 53, "discover": 2, "opened": 4, "looked": 51, "clutched": 2, "hold": 17, "wrist": 1, "hit": 6, "chest": 6, "wasn": 7, "expecting": 1, "ita": 1, "cowardly": 2, "itand": 1, "immediately": 12, "afterwards": 23, "butted": 1, "wrote": 3, "prescription": 2, "folded": 3, "gave": 30, "put": 89, "pocket": 7, "did": 154, "open": 14, "took": 65, "nearest": 2, "chemist": 4, "handed": 4, "didn": 43, "keep": 28, "co": 1, "operative": 1, "stores": 4, "family": 16, "hotel": 14, "combined": 3, "able": 19, "oblige": 1, "hampers": 5, "ran": 7, "1": 5, "lb": 1, "beefsteak": 5, "pt": 1, "bitter": 7, "beer": 10, "6": 2, "hours": 18, "ten": 37, "mile": 25, "morning": 67, "bed": 46, "11": 3, "sharp": 4, "night": 75, "don": 96, "stuff": 6, "things": 67, "understand": 16, "followed": 16, "directions": 5, "resultspeaking": 1, "myselfthat": 1, "preserved": 1, "still": 40, "present": 12, "instance": 5, "beyond": 7, "mistake": 8, "chief": 3, "among": 14, "general": 16, "disinclination": 1, "work": 69, "kind": 16, "suffer": 1, "can": 111, "earliest": 2, "infancy": 1, "martyr": 1, "left": 53, "know": 114, "science": 1, "advanced": 3, "state": 10, "used": 27, "laziness": 2, "skulking": 1, "devil": 1, "say": 71, "something": 37, "living": 15, "give": 41, "pills": 2, "clumps": 2, "strange": 22, "may": 32, "those": 54, "often": 16, "cured": 1, "mefor": 1, "clump": 1, "effect": 6, "make": 53, "anxious": 5, "wanted": 40, "done": 40, "further": 18, "loss": 1, "whole": 40, "box": 3, "does": 44, "sothose": 1, "simple": 16, "fashioned": 2, "remedies": 1, "sometimes": 14, "efficacious": 1, "dispensary": 1, "hour": 38, "describing": 2, "our": 199, "explained": 9, "us": 220, "stood": 20, "hearth": 1, "rug": 4, "clever": 4, "powerful": 3, "piece": 6, "acting": 1, "illustrative": 1, "fancies": 1, "really": 43, "mrs": 18, "poppets": 4, "knocked": 6, "door": 23, "ready": 17, "supper": 37, "smiled": 3, "sadly": 1, "supposed": 4, "better": 27, "try": 28, "swallow": 3, "stomach": 7, "kept": 21, "check": 1, "brought": 17, "tray": 1, "drew": 17, "table": 8, "toyed": 1, "steak": 2, "onions": 3, "rhubarb": 1, "tart": 4, "very": 125, "weak": 10, "interest": 12, "whatever": 9, "foodan": 1, "unusual": 3, "meand": 2, "want": 72, "cheese": 10, "duty": 6, "refilled": 1, "glasses": 3, "lit": 12, "pipes": 8, "resumed": 1, "discussion": 3, "health": 3, "actually": 3, "none": 9, "sure": 19, "unanimous": 1, "itwhatever": 1, "washad": 1, "overwork": 1, "complete": 4, "change": 15, "overstrain": 1, "brains": 1, "has": 96, "produced": 6, "depression": 3, "throughout": 4, "system": 3, "scene": 11, "absence": 3, "necessity": 3, "restore": 1, "mental": 2, "equilibrium": 1, "cousin": 8, "who": 149, "usually": 4, "described": 3, "charge": 7, "sheet": 3, "student": 1, "naturally": 7, "somewhat": 9, "physicianary": 1, "putting": 5, "agreed": 10, "suggested": 15, "seek": 1, "retired": 1, "world": 32, "spot": 21, "madding": 1, "crowd": 13, "dream": 10, "sunny": 9, "drowsy": 2, "lanessome": 1, "forgotten": 6, "nook": 4, "hidden": 1, "fairies": 1, "reach": 14, "noisy": 4, "worldsome": 1, "quaint": 10, "perched": 1, "eyrie": 1, "cliffs": 1, "whence": 1, "surging": 1, "waves": 1, "nineteenth": 5, "century": 8, "sound": 10, "faint": 7, "humpy": 1, "place": 44, "meant": 9, "where": 83, "everybody": 12, "eight": 13, "o": 16, "clock": 12, "couldn": 22, "referee": 1, "love": 11, "money": 7, "miles": 28, "baccy": 1, "beat": 5, "sea": 23, "trip": 24, "objected": 4, "strongly": 3, "couple": 18, "wicked": 6, "monday": 6, "idea": 36, "implanted": 1, "bosom": 6, "enjoy": 6, "yourself": 9, "wave": 4, "airy": 1, "adieu": 1, "boys": 11, "shore": 7, "light": 35, "biggest": 2, "pipe": 11, "swagger": 1, "deck": 4, "captain": 5, "cook": 5, "sir": 22, "francis": 1, "drake": 1, "christopher": 1, "columbus": 1, "rolled": 9, "tuesday": 3, "wish": 11, "wednesday": 2, "thursday": 2, "friday": 4, "dead": 13, "saturday": 6, "beef": 11, "tea": 29, "sit": 29, "answer": 12, "wan": 1, "sweet": 21, "smile": 9, "hearted": 7, "people": 90, "ask": 15, "sunday": 4, "begin": 17, "solid": 1, "food": 6, "bag": 15, "umbrella": 9, "hand": 43, "stand": 23, "gunwale": 3, "waiting": 19, "step": 9, "ashore": 1, "thoroughly": 1, "like": 125, "brother": 6, "law": 5, "short": 10, "once": 57, "benefit": 1, "return": 7, "berth": 1, "london": 12, "liverpool": 7, "sell": 2, "ticket": 4, "offered": 3, "town": 32, "tremendous": 3, "reduction": 1, "eventually": 4, "sold": 3, "eighteenpence": 1, "bilious": 1, "looking": 46, "youth": 2, "advised": 2, "men": 56, "exercise": 7, "pressing": 2, "affectionately": 1, "ll": 30, "enough": 41, "last": 71, "lifetime": 2, "ship": 5, "turning": 10, "somersaults": 1, "dry": 9, "land": 19, "himselfmy": 1, "lawcame": 1, "train": 12, "north": 1, "western": 3, "railway": 7, "fellow": 18, "voyage": 2, "coast": 3, "steward": 3, "whether": 19, "pay": 4, "meal": 6, "arrange": 3, "beforehand": 4, "series": 1, "recommended": 2, "latter": 2, "much": 74, "cheaper": 2, "pounds": 8, "five": 45, "breakfast": 20, "fish": 35, "grill": 1, "lunch": 10, "consisted": 1, "courses": 1, "dinner": 7, "sixsoup": 1, "entree": 1, "joint": 1, "poultry": 1, "salad": 1, "sweets": 1, "dessert": 1, "meat": 7, "friend": 32, "close": 14, "pound": 6, "job": 10, "hearty": 4, "eater": 1, "sheerness": 1, "hungry": 5, "contented": 4, "himself": 59, "boiled": 7, "strawberries": 4, "cream": 5, "deal": 13, "during": 19, "afternoon": 15, "eating": 3, "weeks": 8, "neither": 4, "nor": 10, "eitherseemed": 1, "discontented": 2, "announcement": 1, "aroused": 1, "enthusiasm": 5, "within": 13, "worked": 9, "held": 18, "ropes": 3, "pleasant": 21, "odour": 4, "hot": 7, "ham": 1, "mingled": 4, "fried": 1, "greens": 1, "greeted": 3, "ladder": 4, "oily": 5, "feeble": 4, "reply": 8, "quick": 5, "propped": 1, "leeward": 1, "next": 28, "days": 33, "lived": 8, "blameless": 3, "thin": 4, "biscuits": 4, "soda": 1, "water": 90, "towards": 14, "uppish": 2, "toast": 3, "gorging": 1, "chicken": 2, "broth": 1, "steamed": 1, "landing": 9, "gazed": 4, "regretfully": 1, "she": 107, "goes": 15, "worth": 16, "board": 16, "belongs": 1, "haven": 10, "given": 22, "set": 25, "face": 24, "against": 45, "own": 30, "queer": 2, "afraid": 14, "right": 63, "advise": 2, "think": 62, "both": 38, "always": 47, "mystery": 4, "managed": 8, "sick": 5, "seasaid": 1, "purpose": 8, "affectationsaid": 1, "wished": 10, "anecdotes": 2, "gone": 31, "across": 27, "channel": 2, "rough": 8, "passengers": 3, "tied": 6, "berths": 1, "souls": 1, "second": 12, "mate": 1, "curious": 9, "nobody": 20, "sickon": 1, "plenty": 11, "indeed": 11, "boat": 198, "loads": 1, "met": 23, "yet": 13, "thousands": 4, "sailors": 1, "swarm": 1, "hide": 3, "themselves": 23, "yarmouth": 3, "seeming": 1, "enigma": 1, "easily": 9, "southend": 3, "pier": 3, "recollect": 3, "leaning": 6, "port": 1, "holes": 1, "dangerous": 9, "position": 13, "save": 11, "hi": 7, "shaking": 3, "shoulder": 1, "overboard": 6, "oh": 75, "leave": 13, "coffee": 4, "bath": 10, "voyages": 1, "explaining": 1, "loved": 6, "sailor": 1, "replied": 32, "mild": 2, "young": 57, "envious": 1, "query": 2, "confess": 2, "cape": 1, "horn": 1, "vessel": 1, "wrecked": 1, "weren": 3, "shaky": 1, "thrown": 7, "puzzled": 2, "expression": 18, "yes": 20, "ahyes": 1, "answered": 14, "brightening": 3, "headache": 4, "pickles": 2, "disgraceful": 3, "tasted": 2, "respectable": 12, "discovered": 5, "excellent": 2, "preventive": 1, "sickness": 2, "balancing": 1, "centre": 6, "heaves": 2, "pitches": 1, "move": 5, "body": 10, "rises": 3, "lean": 7, "forward": 4, "till": 27, "almost": 10, "touches": 2, "nose": 22, "end": 42, "gets": 9, "backwards": 1, "balance": 1, "let": 44, "fresh": 9, "air": 30, "quiet": 15, "constant": 1, "occupy": 1, "minds": 4, "including": 4, "hard": 21, "appetite": 4, "sleep": 29, "ought": 23, "tendency": 2, "sleepier": 1, "seeing": 13, "summer": 13, "winter": 3, "alike": 3, "lodging": 4, "suit": 7, "except": 12, "sixpenny": 1, "includes": 1, "bread": 13, "butter": 8, "cake": 3, "ad": 1, "lib": 1, "cheap": 4, "price": 3, "greatly": 2, "credit": 7, "suited": 2, "tone": 8, "imply": 1, "surprised": 13, "sensible": 4, "struck": 10, "suggestion": 4, "care": 21, "fellows": 13, "says": 31, "scenery": 7, "line": 32, "smoke": 3, "rat": 3, "won": 11, "stop": 31, "fooling": 5, "slop": 1, "bally": 4, "foolishness": 2, "ii": 3, "plans": 4, "discussed": 7, "pleasures": 4, "camping": 7, "fine": 23, "nights": 4, "ditto": 1, "wet": 21, "compromise": 3, "decided": 11, "impressions": 1, "fears": 3, "lest": 2, "subsequently": 2, "dismissed": 2, "groundless": 1, "meeting": 4, "adjourns": 1, "maps": 1, "arranged": 4, "following": 11, "kingston": 17, "chertsey": 1, "city": 8, "bank": 53, "saturdays": 1, "wake": 17, "outside": 12, "meet": 13, "camp": 5, "inns": 6, "wild": 11, "free": 4, "patriarchal": 1, "slowly": 13, "golden": 6, "memory": 2, "sun": 8, "fades": 1, "hearts": 3, "cold": 30, "sad": 19, "clouds": 4, "silent": 11, "sorrowing": 1, "birds": 2, "ceased": 1, "song": 31, "moorhen": 1, "plaintive": 1, "cry": 11, "harsh": 2, "croak": 1, "corncrake": 1, "stirs": 1, "awed": 2, "hush": 2, "around": 16, "couch": 1, "waters": 10, "dying": 4, "breathes": 2, "her": 99, "dim": 6, "woods": 11, "either": 17, "ghostly": 3, "army": 1, "grey": 6, "shadows": 7, "creep": 3, "noiseless": 2, "tread": 1, "chase": 1, "lingering": 1, "rear": 4, "guard": 2, "unseen": 1, "feet": 18, "above": 23, "waving": 4, "grass": 13, "sighing": 1, "rushes": 5, "sombre": 2, "throne": 1, "folds": 1, "black": 11, "wings": 4, "darkening": 1, "phantom": 1, "palace": 7, "pale": 1, "stars": 6, "reigns": 1, "stillness": 4, "run": 14, "tent": 8, "pitched": 3, "frugal": 1, "cooked": 1, "eaten": 2, "big": 13, "filled": 5, "lighted": 2, "chat": 5, "musical": 6, "undertone": 1, "pauses": 1, "talk": 7, "playing": 8, "prattles": 1, "tales": 5, "secrets": 1, "sings": 5, "low": 14, "child": 9, "sung": 4, "many": 37, "thousand": 5, "yearswill": 1, "sing": 17, "voice": 17, "grows": 9, "olda": 1, "changing": 1, "nestled": 2, "yielding": 1, "though": 20, "mere": 12, "words": 7, "story": 10, "listen": 5, "margin": 3, "moon": 3, "loves": 2, "stoops": 1, "kiss": 3, "sister": 5, "throws": 3, "silver": 4, "arms": 15, "clingingly": 1, "flows": 1, "singing": 14, "whispering": 2, "king": 17, "seatill": 1, "voices": 8, "die": 8, "silence": 17, "outtill": 1, "common": 11, "everyday": 1, "strangely": 3, "full": 30, "thoughts": 9, "speaktill": 1, "laugh": 13, "rising": 8, "knock": 4, "ashes": 2, "burnt": 2, "lulled": 2, "lapping": 2, "rustling": 2, "trees": 9, "fall": 7, "asleep": 7, "beneath": 7, "great": 62, "againyoung": 1, "ere": 5, "centuries": 3, "fret": 2, "furrowed": 1, "fair": 15, "sins": 1, "follies": 1, "loving": 5, "heartsweet": 1, "bygone": 2, "new": 20, "mother": 17, "nursed": 1, "breastere": 1, "wiles": 1, "painted": 2, "civilization": 1, "lured": 2, "fond": 6, "poisoned": 1, "sneers": 1, "artificiality": 1, "ashamed": 4, "led": 4, "stately": 2, "home": 31, "mankind": 3, "ago": 18, "rained": 1, "rouse": 2, "poetry": 1, "harrisno": 1, "yearning": 2, "unattainable": 2, "weeps": 1, "knows": 9, "eyes": 22, "fill": 1, "tears": 7, "bet": 3, "raw": 2, "worcester": 1, "chop": 2, "mermaid": 1, "hark": 1, "mermaids": 1, "below": 11, "spirits": 1, "chanting": 1, "dirges": 1, "white": 13, "corpses": 3, "seaweed": 2, "arm": 4, "ve": 31, "chill": 2, "along": 31, "corner": 13, "here": 51, "drop": 14, "finest": 1, "scotch": 5, "whisky": 9, "tastedput": 1, "brilliant": 5, "drinking": 3, "believe": 14, "paradise": 1, "supposing": 2, "likely": 4, "greet": 2, "glad": 6, "found": 37, "nice": 6, "nectar": 1, "regarded": 7, "practical": 5, "timely": 1, "hint": 2, "rainy": 1, "evening": 36, "inches": 4, "damp": 8, "banks": 9, "puddly": 1, "places": 7, "seen": 25, "lug": 3, "proceed": 1, "fix": 7, "soaked": 3, "heavy": 13, "flops": 1, "tumbles": 2, "clings": 1, "makes": 19, "mad": 13, "rain": 22, "pouring": 4, "steadily": 13, "difficult": 10, "task": 2, "becomes": 6, "herculean": 1, "instead": 12, "helping": 3, "simply": 8, "fool": 8, "beautifully": 5, "fixed": 15, "gives": 7, "hoist": 1, "spoils": 3, "retorts": 1, "leggo": 1, "pull": 20, "wrong": 19, "stupid": 4, "ass": 7, "shout": 14, "yells": 1, "roar": 4, "wishing": 2, "pulls": 1, "pegs": 1, "ah": 15, "idiot": 3, "mutter": 3, "comes": 16, "savage": 4, "haul": 4, "lay": 13, "mallet": 1, "business": 14, "same": 44, "starts": 2, "direction": 5, "explain": 4, "views": 4, "follow": 12, "swearing": 4, "until": 36, "heap": 5, "ruins": 5, "indignantly": 6, "exclaim": 3, "breath": 3, "meanwhile": 2, "third": 12, "baling": 1, "spilled": 1, "sleeve": 3, "cursing": 3, "minutes": 40, "thundering": 1, "blazes": 1, "re": 20, "blarmed": 1, "isn": 8, "hopeless": 1, "attempting": 1, "wood": 13, "fire": 10, "methylated": 6, "spirit": 16, "stove": 9, "rainwater": 2, "article": 5, "diet": 3, "thirds": 1, "pie": 18, "exceedingly": 2, "rich": 9, "jam": 9, "salt": 2, "soup": 2, "tobacco": 2, "luckily": 1, "bottle": 5, "cheers": 2, "inebriates": 1, "taken": 29, "proper": 11, "quantity": 3, "restores": 1, "sufficient": 10, "induce": 1, "elephant": 2, "suddenly": 16, "volcano": 1, "exploded": 2, "seathe": 1, "sleeping": 6, "peacefully": 1, "grasp": 3, "terrible": 2, "happened": 26, "impression": 4, "thieves": 1, "murderers": 1, "express": 3, "usual": 3, "method": 10, "help": 18, "kicking": 3, "smothered": 4, "somebody": 19, "trouble": 18, "cries": 2, "coming": 21, "underneath": 6, "determining": 1, "events": 6, "dearly": 2, "struggle": 5, "frantically": 2, "hitting": 1, "legs": 16, "yelling": 2, "lustily": 2, "dimly": 2, "observe": 2, "dressed": 8, "ruffian": 1, "kill": 5, "preparing": 1, "death": 12, "begins": 4, "dawn": 2, "jim": 5, "recognising": 1, "rubbing": 1, "blown": 1, "bill": 7, "raise": 1, "ground": 10, "rocks": 3, "muffled": 1, "heard": 30, "replies": 4, "ruin": 1, "struggles": 3, "muddy": 2, "trampled": 1, "unnecessarily": 2, "aggressive": 1, "moodhe": 1, "under": 45, "evident": 7, "belief": 2, "speechless": 1, "owing": 2, "having": 29, "caught": 37, "colds": 3, "also": 22, "quarrelsome": 2, "swear": 3, "hoarse": 1, "whispers": 2, "therefore": 9, "inn": 14, "pub": 3, "folks": 2, "inclined": 4, "hailed": 3, "approval": 1, "revel": 4, "romantic": 1, "solitude": 2, "trifle": 3, "jollier": 1, "imagine": 4, "angel": 4, "sent": 12, "earth": 11, "reason": 7, "withheld": 1, "shape": 3, "small": 24, "fox": 6, "terrier": 5, "nobler": 1, "bring": 12, "pious": 2, "ladies": 8, "gentlemen": 11, "expense": 2, "dog": 37, "snatched": 5, "skies": 1, "chariot": 1, "happen": 7, "paid": 3, "dozen": 10, "chickens": 1, "killed": 8, "dragged": 4, "growling": 3, "scruff": 1, "neck": 5, "fourteen": 2, "street": 17, "fights": 2, "cat": 19, "inspection": 2, "irate": 1, "female": 3, "called": 17, "murderer": 3, "summoned": 1, "ferocious": 1, "large": 9, "pinned": 1, "tool": 1, "shed": 1, "venture": 1, "learned": 6, "gardener": 1, "unknown": 4, "thirty": 13, "shillings": 8, "backing": 3, "rats": 4, "maybe": 5, "d": 24, "remain": 1, "longer": 9, "hang": 13, "stable": 1, "collect": 2, "gang": 1, "disreputable": 4, "dogs": 24, "lead": 2, "march": 7, "slums": 1, "fight": 11, "observed": 3, "pubs": 1, "hotels": 4, "emphatic": 1, "approbation": 1, "thus": 9, "settled": 11, "arrangements": 6, "satisfaction": 3, "discuss": 3, "begun": 3, "argue": 2, "oratory": 1, "proposed": 3, "saying": 3, "square": 6, "irish": 7, "glass": 15, "thirsty": 4, "presentiment": 1, "warm": 5, "slice": 1, "lemon": 1, "debate": 1, "assent": 1, "adjourned": 2, "assembly": 1, "hats": 4, "iii": 1, "elderly": 3, "puts": 3, "remark": 2, "delights": 1, "early": 29, "bathing": 8, "provisions": 3, "upset": 17, "assembled": 2, "settle": 2, "paper": 10, "write": 5, "j": 6, "grocery": 1, "catalogue": 2, "pencil": 2, "overso": 1, "burden": 1, "backs": 4, "reminds": 4, "poor": 16, "uncle": 11, "podger": 12, "commotion": 2, "house": 47, "undertook": 1, "frame": 2, "maker": 2, "standing": 10, "dining": 1, "aunt": 5, "worry": 2, "yourselves": 2, "coat": 9, "send": 5, "girl": 6, "sixpen": 1, "orth": 1, "nails": 2, "size": 5, "gradually": 1, "candle": 5, "hammer": 13, "rule": 13, "tom": 17, "kitchen": 3, "chair": 17, "mr": 8, "goggles": 1, "pa": 1, "regards": 2, "hopes": 2, "leg": 6, "lend": 2, "level": 4, "maria": 4, "cord": 1, "lift": 1, "cut": 7, "spring": 8, "handkerchief": 7, "tools": 2, "dance": 2, "hinder": 2, "etc": 1, "doesn": 9, "anybody": 13, "lifeupon": 1, "word": 17, "expect": 10, "spent": 7, "tying": 2, "finger": 2, "charwoman": 3, "semi": 1, "circle": 1, "fourth": 2, "nail": 9, "fifth": 1, "injured": 3, "knees": 3, "grovel": 1, "grunt": 2, "lost": 14, "heavens": 2, "gaping": 1, "sight": 29, "mark": 3, "wall": 16, "beside": 2, "different": 12, "fools": 2, "measure": 3, "eighths": 1, "heads": 4, "arrive": 3, "results": 1, "sneer": 1, "row": 15, "number": 18, "use": 13, "string": 6, "critical": 1, "angle": 1, "trying": 22, "possible": 6, "slip": 4, "slide": 1, "piano": 5, "suddenness": 1, "notes": 2, "allow": 6, "language": 14, "blow": 8, "smash": 2, "thumb": 1, "yell": 4, "toes": 1, "mildly": 1, "hoped": 5, "spend": 2, "women": 6, "fuss": 4, "picking": 2, "admiring": 2, "clean": 8, "plaster": 2, "precipitated": 1, "force": 3, "nearly": 25, "flatten": 1, "hole": 4, "midnight": 4, "upvery": 1, "crooked": 1, "insecure": 1, "yards": 15, "smoothed": 1, "rake": 1, "wretchedexcept": 1, "stepping": 1, "heavily": 2, "corns": 1, "surveying": 1, "mess": 5, "pride": 9, "permit": 1, "labour": 5, "discarded": 2, "clear": 11, "upper": 2, "reaches": 4, "thames": 21, "navigation": 1, "sufficiently": 4, "indispensable": 1, "tore": 2, "track": 1, "altogether": 15, "downright": 1, "wisdom": 1, "merely": 11, "reference": 1, "load": 1, "danger": 6, "swamping": 1, "store": 1, "foolish": 3, "essential": 1, "pleasure": 6, "comfort": 5, "useless": 5, "lumber": 5, "pile": 1, "craft": 7, "mast": 10, "high": 19, "clothes": 19, "houses": 10, "servants": 1, "host": 1, "swell": 3, "friends": 23, "twopence": 5, "ha": 7, "pence": 1, "expensive": 2, "entertainments": 1, "enjoys": 1, "formalities": 1, "fashions": 1, "pretence": 1, "ostentation": 1, "withoh": 1, "heaviest": 1, "maddest": 1, "dread": 1, "neighbour": 1, "luxuries": 1, "cloy": 2, "bore": 1, "empty": 9, "show": 15, "criminal": 1, "iron": 8, "crown": 7, "yore": 1, "bleed": 2, "swoon": 1, "aching": 2, "wears": 1, "manall": 1, "throw": 8, "oars": 2, "cumbersome": 1, "manage": 3, "freedom": 2, "anxiety": 5, "dreamy": 2, "lazinessno": 1, "windy": 3, "skimming": 1, "lightly": 3, "er": 2, "shallows": 2, "glittering": 3, "sunbeams": 1, "flitting": 1, "ripples": 1, "image": 1, "green": 9, "lilies": 2, "yellow": 4, "sedges": 1, "orchis": 1, "blue": 12, "nots": 1, "packed": 12, "needa": 1, "homely": 1, "name": 8, "someone": 5, "eat": 8, "wear": 1, "drink": 15, "thirst": 2, "easier": 3, "liable": 2, "plain": 6, "merchandise": 1, "sunshinetime": 1, "\\u00e6olian": 1, "music": 11, "wind": 27, "god": 6, "draws": 2, "human": 13, "strings": 1, "ustime": 1, "beg": 7, "pardon": 5, "forgot": 5, "cover": 5, "simpler": 3, "comfortable": 6, "adopted": 2, "hoops": 6, "stretch": 9, "huge": 8, "canvas": 14, "fasten": 1, "stem": 1, "stern": 10, "converts": 1, "cosy": 2, "stuffy": 2, "drawbacks": 1, "died": 12, "funeral": 3, "expenses": 1, "lamp": 4, "soap": 4, "brush": 7, "comb": 3, "between": 28, "toothbrush": 1, "basin": 2, "tooth": 7, "powder": 1, "shaving": 3, "tackle": 2, "sounds": 3, "french": 9, "towels": 3, "notice": 13, "gigantic": 2, "anywhere": 7, "near": 24, "bathe": 5, "determinewhen": 1, "thinking": 12, "londonthat": 1, "dip": 2, "religiously": 3, "pack": 7, "pair": 11, "drawers": 4, "towel": 4, "red": 17, "complexion": 2, "contrary": 5, "twice": 7, "virtue": 2, "triumphed": 1, "stumbled": 1, "dismally": 1, "enjoyed": 5, "seem": 21, "specially": 2, "cutting": 2, "east": 2, "pick": 3, "cornered": 1, "stones": 4, "top": 19, "sharpen": 1, "points": 1, "sand": 3, "huddle": 1, "hop": 2, "shivering": 1, "insulting": 3, "catches": 2, "chucks": 1, "posture": 1, "rock": 1, "ugh": 1, "carries": 2, "mid": 7, "ocean": 2, "strike": 2, "wonder": 10, "kinder": 1, "hope": 11, "retires": 2, "sprawling": 4, "star": 1, "swimming": 2, "dress": 13, "crawl": 1, "pretend": 2, "liked": 8, "talked": 4, "swim": 3, "plunge": 2, "limpid": 1, "ordinarily": 2, "ate": 5, "protest": 1, "towing": 12, "stream": 31, "urged": 3, "pleasanter": 1, "even": 36, "few": 27, "hundredweight": 1, "withdrew": 2, "opposition": 2, "finally": 5, "suits": 2, "flannel": 1, "wash": 7, "ourselves": 26, "dirty": 7, "asked": 44, "washing": 9, "flannels": 2, "easy": 13, "influence": 2, "experience": 5, "shirts": 1, "trousers": 8, "learn": 2, "late": 12, "miserable": 5, "impostor": 2, "these": 37, "afterbut": 1, "shilling": 3, "shockers": 1, "anticipate": 1, "impressed": 2, "socks": 3, "handkerchiefs": 2, "wipe": 2, "leather": 3, "boots": 11, "boating": 15, "shoes": 5, "iv": 1, "question": 11, "objections": 1, "paraffine": 6, "oil": 14, "atmosphere": 3, "advantages": 2, "travelling": 4, "companion": 5, "married": 8, "woman": 17, "deserts": 1, "provision": 1, "cussedness": 3, "brushes": 2, "awful": 10, "behaviour": 3, "retire": 1, "frying": 9, "pan": 11, "indigestible": 2, "pot": 4, "kettle": 16, "significant": 1, "shop": 8, "oozed": 4, "ooze": 1, "rudder": 4, "impregnating": 1, "saturated": 1, "spoilt": 1, "westerly": 1, "blew": 4, "easterly": 1, "northerly": 1, "southerly": 1, "arctic": 1, "snows": 1, "raised": 2, "waste": 1, "desert": 4, "sands": 1, "laden": 2, "fragrance": 1, "ruined": 6, "sunset": 4, "moonbeams": 1, "positively": 1, "reeked": 1, "marlow": 17, "bridge": 18, "escape": 3, "passed": 19, "church": 15, "yard": 6, "buried": 6, "stunk": 1, "birmingham": 1, "country": 11, "steeped": 1, "together": 22, "lonely": 4, "field": 6, "blasted": 1, "oak": 13, "oath": 3, "middle": 27, "affair": 3, "confined": 1, "wholesome": 1, "quantities": 1, "eggs": 12, "bacon": 6, "jambut": 1, "itself": 21, "hamper": 17, "cheesy": 1, "flavour": 2, "apple": 4, "german": 16, "sausage": 2, "buying": 1, "cheeses": 15, "splendid": 2, "ripe": 1, "mellow": 2, "horse": 5, "power": 2, "scent": 1, "warranted": 1, "carry": 2, "mind": 31, "cab": 5, "ramshackle": 1, "kneed": 1, "broken": 13, "winded": 2, "somnambulist": 1, "owner": 4, "conversation": 10, "referred": 2, "shamble": 1, "swiftest": 1, "steam": 19, "roller": 1, "built": 6, "merry": 5, "bell": 3, "whiff": 1, "steed": 2, "woke": 11, "snort": 1, "terror": 1, "dashed": 3, "reached": 19, "laying": 1, "rate": 7, "leaving": 5, "cripples": 1, "stout": 4, "nowhere": 1, "porters": 1, "driver": 2, "station": 7, "presence": 6, "brown": 6, "marched": 1, "proudly": 1, "platform": 6, "falling": 6, "respectfully": 1, "crowded": 4, "carriage": 5, "already": 7, "crusty": 1, "gentleman": 14, "notwithstanding": 2, "rack": 1, "squeezed": 1, "moments": 5, "fidget": 1, "oppressive": 2, "sniffing": 1, "sniff": 3, "rose": 8, "lady": 23, "harried": 1, "gathered": 7, "parcels": 2, "remaining": 1, "solemn": 6, "appearance": 6, "belong": 1, "undertaker": 1, "baby": 5, "laughed": 8, "pleasantly": 3, "depressed": 1, "crewe": 2, "accepted": 1, "forced": 3, "buffet": 1, "yelled": 4, "stamped": 1, "waved": 2, "umbrellas": 3, "quarter": 12, "yours": 8, "brandy": 2, "neat": 3, "please": 8, "miss": 4, "responded": 1, "quietly": 9, "drunk": 3, "compartment": 1, "stations": 1, "rush": 8, "y": 1, "carrying": 9, "bags": 6, "mount": 1, "steps": 5, "stagger": 1, "behind": 29, "droop": 1, "squeeze": 1, "carriages": 1, "difference": 3, "euston": 1, "wife": 8, "smelt": 2, "instant": 6, "worst": 2, "bought": 4, "added": 13, "understood": 1, "speak": 8, "detained": 1, "later": 14, "returned": 6, "directed": 2, "moist": 1, "touch": 8, "attached": 4, "queried": 4, "sovereign": 2, "bury": 5, "keeping": 7, "madam": 1, "smell": 4, "journey": 5, "ending": 1, "holiday": 4, "consider": 2, "others": 8, "whose": 11, "roof": 4, "honour": 3, "residing": 1, "widow": 3, "possibly": 5, "orphan": 2, "strong": 13, "eloquent": 2, "terms": 2, "husband": 4, "instinctively": 2, "regard": 5, "decline": 3, "detect": 1, "melons": 1, "argued": 2, "injury": 1, "result": 9, "fifteen": 3, "guineas": 1, "reckoning": 1, "cost": 3, "sixpence": 2, "means": 2, "rid": 5, "threw": 10, "canal": 2, "bargemen": 1, "complained": 3, "dark": 22, "parish": 2, "mortuary": 1, "coroner": 1, "plot": 1, "deprive": 1, "waking": 2, "taking": 15, "burying": 1, "beach": 3, "gained": 2, "reputation": 1, "visitors": 2, "noticed": 13, "chested": 1, "consumptive": 1, "throng": 1, "declining": 1, "shan": 2, "fell": 9, "slap": 4, "sevendinner": 1, "cheerful": 7, "fruit": 3, "pies": 4, "tomatoes": 2, "wonderful": 6, "sticky": 1, "concoction": 1, "mixed": 5, "lemonade": 5, "harped": 1, "wine": 1, "sleepy": 2, "mouch": 3, "girls": 17, "blazing": 1, "pretty": 26, "lengthy": 1, "parted": 1, "gladstone": 5, "victuals": 1, "cooking": 3, "utensils": 2, "moved": 2, "window": 8, "piled": 4, "floor": 5, "packing": 5, "person": 6, "surprises": 3, "subjects": 1, "entirely": 5, "readiness": 3, "uncanny": 3, "spread": 7, "cocked": 1, "cigar": 2, "intended": 3, "boss": 1, "potter": 2, "pushing": 1, "aside": 5, "teaching": 1, "irritated": 2, "irritate": 3, "working": 6, "loll": 1, "sofa": 1, "wherever": 1, "real": 15, "messing": 1, "idle": 1, "gaped": 1, "yawned": 2, "noble": 9, "slaving": 1, "superintend": 1, "hands": 8, "pockets": 3, "energetic": 3, "nature": 20, "strapped": 3, "ain": 9, "laughedone": 1, "irritating": 5, "senseless": 2, "chuckle": 5, "headed": 7, "crack": 1, "jawed": 1, "laughs": 1, "horrible": 1, "occurred": 3, "haunts": 1, "misery": 3, "perspiration": 1, "hunt": 1, "unpack": 2, "repack": 1, "upstairs": 5, "wrapped": 7, "boot": 6, "mortal": 2, "rummaged": 1, "created": 1, "chaos": 1, "reigned": 1, "eighteen": 6, "shook": 4, "inside": 14, "repacked": 1, "slammed": 1, "pouch": 1, "10": 4, "5": 3, "p": 3, "remained": 2, "wanting": 1, "twelve": 8, "intending": 1, "comment": 2, "waited": 10, "hanged": 1, "packer": 1, "piles": 1, "plates": 4, "cups": 3, "kettles": 2, "bottles": 2, "jars": 1, "stoves": 1, "cakes": 2, "c": 3, "soon": 21, "become": 14, "exciting": 9, "breaking": 5, "cup": 6, "strawberry": 1, "tomato": 2, "squashed": 1, "teaspoon": 2, "trod": 2, "edge": 7, "watched": 9, "excited": 4, "stepped": 7, "smashed": 2, "slipper": 2, "wouldn": 7, "scrape": 2, "staring": 1, "seat": 8, "stared": 3, "mysterious": 4, "exclaimed": 7, "cried": 9, "spinning": 1, "roared": 9, "flying": 4, "teapot": 1, "ambition": 2, "sworn": 3, "squirm": 1, "particularly": 7, "perfect": 6, "nuisance": 1, "wasted": 4, "stumble": 1, "curse": 2, "highest": 1, "aim": 2, "object": 5, "succeeded": 6, "accomplishing": 2, "conceit": 1, "unbearable": 1, "laboured": 1, "whenever": 7, "worried": 1, "teaspoons": 1, "pretended": 3, "lemons": 1, "encouraged": 1, "encourage": 3, "encouragement": 4, "natural": 5, "sin": 4, "12": 1, "50": 1, "reflection": 1, "tossed": 3, "beds": 7, "prefer": 4, "preferred": 5, "nosix": 1, "split": 1, "past": 31, "30": 1, "placed": 2, "tumble": 2, "luggage": 7, "v": 1, "arouses": 1, "sluggard": 1, "forecast": 4, "swindle": 1, "depravity": 1, "gather": 3, "drive": 3, "style": 10, "waterloo": 4, "innocence": 1, "south": 4, "officials": 1, "concerning": 3, "worldly": 2, "trains": 1, "afloat": 2, "nine": 12, "starting": 4, "keyhole": 1, "oversleeping": 1, "retorted": 3, "um": 1, "lucky": 3, "lain": 2, "snoring": 1, "snarled": 1, "strain": 3, "interrupted": 1, "defiant": 1, "snore": 1, "reminded": 3, "existence": 3, "laythe": 1, "uson": 1, "mouth": 4, "wide": 3, "maddens": 1, "shocking": 1, "precious": 1, "lifethe": 1, "priceless": 2, "againbeing": 1, "brutish": 1, "throwing": 5, "hideous": 2, "sloth": 1, "inestimable": 1, "gift": 1, "valuable": 1, "hereafter": 1, "passing": 12, "unused": 1, "stuffing": 1, "flirting": 1, "slavey": 1, "sunk": 4, "soul": 11, "clogging": 1, "oblivion": 1, "appeared": 8, "resolve": 1, "dispute": 2, "flew": 6, "slung": 1, "landed": 6, "shouted": 7, "ear": 4, "awoke": 2, "wasermarrer": 1, "fat": 3, "chunk": 2, "shrieked": 2, "jumping": 1, "thunder": 3, "dressing": 3, "extras": 1, "remembered": 2, "downstairs": 5, "anyone": 4, "absurd": 3, "certainly": 7, "cared": 2, "vulgar": 2, "lump": 1, "invited": 4, "whiling": 1, "fighting": 7, "doorstep": 3, "calmed": 1, "chops": 2, "wait": 11, "fatalities": 1, "prophesied": 1, "ghastly": 4, "occasional": 3, "local": 6, "storms": 1, "midland": 1, "counties": 1, "bar": 5, "silly": 7, "tomfoolishness": 1, "plagued": 1, "fraud": 1, "aggravating": 1, "forecasts": 1, "precisely": 3, "yesterday": 4, "opposite": 15, "completely": 3, "autumn": 1, "paying": 4, "attention": 5, "report": 1, "newspaper": 2, "showers": 2, "thunderstorms": 2, "picnic": 7, "indoors": 1, "wagonettes": 1, "coaches": 1, "jolly": 10, "shining": 1, "cloud": 2, "chuckled": 1, "stirred": 2, "books": 1, "specimens": 1, "cockle": 1, "shells": 1, "heat": 3, "became": 5, "lark": 1, "landlady": 3, "lovely": 14, "wetno": 1, "sign": 3, "cheer": 1, "shelter": 3, "drenched": 1, "grand": 9, "flimsy": 1, "bitterly": 2, "rheumatism": 1, "barometer": 2, "misleading": 2, "hanging": 2, "oxford": 13, "staying": 1, "pointing": 2, "matters": 7, "tapped": 3, "jumped": 3, "pointed": 7, "morrow": 2, "fancied": 4, "higher": 3, "faster": 1, "pointer": 1, "peg": 2, "best": 17, "instrument": 6, "prophesy": 1, "harder": 4, "prognosticate": 1, "drought": 2, "famine": 1, "sunstroke": 2, "simooms": 1, "prevented": 1, "content": 2, "steady": 6, "torrent": 1, "lower": 5, "part": 28, "overflowed": 1, "prolonged": 1, "spell": 1, "poem": 3, "printed": 1, "oracle": 1, "foretold": 2, "machine": 3, "referring": 1, "barometers": 1, "ones": 3, "tail": 8, "falls": 2, "nly": 1, "ely": 2, "tap": 3, "correct": 3, "reduce": 1, "fahrenheit": 1, "prophet": 1, "gloomy": 5, "horizon": 1, "break": 10, "affection": 1, "lessened": 1, "circumstances": 2, "clearing": 1, "continuing": 2, "prophesies": 1, "entertain": 1, "revengeful": 1, "ye": 5, "cheerily": 5, "portent": 1, "proves": 1, "angry": 2, "vague": 2, "notion": 4, "especial": 1, "blood": 7, "curdling": 3, "readings": 1, "atmospheric": 1, "disturbance": 1, "oblique": 1, "southern": 1, "europe": 2, "pressure": 1, "increasing": 1, "finding": 5, "wretched": 6, "wasting": 2, "sneaked": 2, "cigarette": 1, "carefully": 6, "carted": 1, "roll": 3, "rugs": 9, "overcoats": 1, "macintoshes": 1, "melon": 1, "bulky": 1, "grapes": 2, "japanese": 2, "lot": 8, "apparently": 4, "biggs": 11, "greengrocer": 2, "talent": 1, "lies": 7, "securing": 1, "services": 2, "abandoned": 3, "unprincipled": 1, "errand": 1, "civilisation": 1, "villainous": 1, "crops": 2, "neighbourhood": 11, "latest": 1, "coram": 1, "murder": 3, "promptly": 5, "period": 4, "cross": 4, "examination": 1, "subjected": 1, "19": 1, "orders": 1, "crime": 2, "assisted": 2, "21": 2, "prove": 2, "alibi": 2, "importance": 1, "hurry": 8, "dawned": 2, "vision": 8, "catching": 4, "eased": 4, "frowned": 1, "wounded": 1, "sensitive": 2, "touchy": 1, "railings": 1, "selecting": 1, "straw": 3, "chew": 1, "grocer": 5, "42": 1, "moving": 5, "joined": 8, "superintendent": 2, "posts": 5, "independent": 1, "curb": 1, "starve": 1, "atlantic": 2, "stanley": 1, "collected": 2, "asking": 4, "party": 16, "giddy": 1, "portion": 1, "wedding": 1, "bridegroom": 1, "elder": 3, "thoughtful": 8, "populace": 1, "probably": 3, "corpse": 3, "cabs": 1, "belongings": 1, "shooting": 1, "forsake": 1, "drove": 3, "amidst": 2, "shying": 1, "carrot": 1, "luck": 5, "eleven": 5, "porter": 5, "whom": 9, "rumour": 2, "master": 3, "convinced": 3, "traffic": 1, "authorities": 1, "southampton": 1, "windsor": 7, "loop": 2, "engine": 1, "anyhow": 5, "confident": 2, "9": 1, "32": 3, "virginia": 1, "isle": 1, "wight": 1, "somewhere": 7, "slipped": 8, "begged": 2, "gents": 1, "suppose": 12, "gimme": 2, "exeter": 1, "mail": 1, "wended": 2, "stored": 1, "sculls": 14, "tiller": 1, "lines": 19, "unhappy": 1, "deeply": 2, "suspicious": 1, "prow": 3, "shot": 5, "vi": 1, "instructive": 2, "remarks": 4, "english": 8, "history": 5, "observations": 1, "carved": 10, "stivvings": 3, "junior": 1, "musings": 1, "antiquity": 1, "steering": 9, "hampton": 10, "court": 7, "maze": 8, "guide": 6, "glorious": 7, "dainty": 7, "sheen": 1, "leaf": 2, "blushing": 2, "deeper": 2, "year": 8, "maid": 1, "trembling": 2, "wakening": 1, "pulses": 1, "brink": 2, "womanhood": 1, "streets": 4, "picturesque": 10, "flashing": 3, "sunlight": 7, "glinting": 2, "drifting": 3, "barges": 6, "wooded": 5, "towpath": 2, "trim": 3, "villas": 1, "orange": 4, "blazer": 4, "grunting": 1, "distant": 5, "glimpses": 2, "tudors": 2, "calm": 8, "peaceful": 8, "dreamily": 2, "musing": 2, "fit": 4, "mused": 2, "kyningestun": 2, "saxon": 6, "kinges": 1, "crowned": 1, "c\\u00e6sar": 9, "crossed": 2, "roman": 7, "legions": 2, "camped": 1, "sloping": 1, "uplands": 1, "elizabeth": 3, "everywhere": 1, "queen": 8, "bess": 1, "public": 4, "nuts": 1, "england": 5, "virgin": 1, "scarcely": 1, "attractions": 2, "slept": 11, "prime": 1, "minister": 1, "signs": 3, "patronised": 2, "88": 1, "chucked": 3, "december": 1, "1886": 1, "entered": 2, "famous": 8, "flock": 1, "minded": 6, "edwy": 2, "hated": 1, "coronation": 1, "feast": 2, "boar": 2, "stuffed": 4, "sugar": 1, "plums": 1, "sack": 1, "mead": 1, "steal": 3, "moonlight": 5, "beloved": 3, "elgiva": 1, "perhaps": 7, "casement": 1, "watching": 4, "halls": 1, "boisterous": 1, "revelry": 2, "floated": 1, "bursts": 2, "din": 2, "tumult": 1, "brutal": 3, "odo": 1, "dunstan": 1, "rude": 7, "hurl": 1, "coarse": 1, "insults": 1, "faced": 2, "drag": 3, "loud": 3, "clamour": 3, "drunken": 2, "brawl": 1, "crash": 2, "battle": 6, "kings": 2, "greatness": 1, "rise": 2, "stuarts": 1, "royal": 2, "strained": 2, "moorings": 1, "cloaked": 2, "gallants": 1, "swaggered": 1, "ferry": 1, "ho": 1, "gadzooks": 1, "gramercy": 1, "plainly": 1, "borough": 1, "nobles": 1, "courtiers": 1, "road": 10, "gates": 6, "gay": 5, "clanking": 1, "steel": 2, "prancing": 2, "palfreys": 1, "silks": 2, "velvets": 1, "faces": 9, "spacious": 1, "oriel": 1, "latticed": 3, "windows": 6, "fireplaces": 1, "gabled": 1, "roofs": 1, "breathe": 1, "hose": 1, "doublet": 1, "pearl": 1, "embroidered": 1, "stomachers": 1, "complicated": 2, "oaths": 3, "upraised": 1, "build": 4, "bricks": 2, "grown": 4, "firmly": 7, "stairs": 5, "creak": 2, "speaking": 6, "staircases": 1, "magnificent": 1, "staircase": 3, "market": 3, "mansion": 1, "personage": 1, "lives": 5, "buy": 2, "hat": 12, "thoughtless": 3, "shopman": 2, "staggered": 2, "quickly": 7, "recovering": 3, "hero": 2, "thereupon": 3, "balusters": 1, "superb": 1, "workmanship": 1, "panelled": 3, "carving": 3, "drawing": 7, "decorated": 1, "startling": 1, "remarkable": 7, "apartment": 1, "proprietor": 4, "forth": 5, "wooden": 3, "ceiling": 4, "expostulated": 1, "covered": 5, "match": 1, "blame": 5, "doubtless": 2, "relief": 4, "average": 4, "householder": 1, "desiring": 1, "curiosity": 2, "maniac": 2, "doubt": 5, "depressing": 2, "lie": 12, "enormous": 1, "prices": 2, "wives": 1, "single": 2, "couples": 2, "childless": 1, "lovers": 7, "bother": 4, "smith": 1, "marry": 1, "dwell": 1, "school": 9, "sandford": 4, "merton": 3, "lad": 4, "rows": 4, "greek": 1, "irregular": 1, "verbs": 1, "weird": 6, "unnatural": 1, "notions": 2, "parents": 3, "yearned": 3, "win": 1, "prizes": 1, "grow": 4, "sorts": 1, "ideas": 3, "creature": 1, "harmless": 2, "babe": 1, "unborn": 2, "badly": 5, "bronchitis": 2, "hay": 1, "christmas": 1, "stricken": 1, "rheumatic": 2, "november": 1, "fog": 2, "laughing": 14, "gas": 7, "teeth": 4, "false": 2, "suffered": 2, "terribly": 3, "toothache": 1, "neuralgia": 1, "ache": 1, "chilblains": 1, "scare": 1, "1871": 1, "singularly": 3, "reputed": 1, "custards": 1, "sob": 2, "latin": 1, "exercises": 1, "grammar": 1, "sacrificed": 1, "sake": 2, "desire": 8, "excuse": 2, "catch": 11, "stiff": 4, "fooled": 1, "draughts": 1, "freshened": 1, "holidays": 1, "whooping": 1, "cough": 1, "kinds": 2, "disorders": 1, "lasted": 1, "term": 2, "recommenced": 1, "spite": 5, "man\\u0153uvre": 1, "oven": 1, "baked": 2, "artistic": 1, "beautiful": 14, "grandfathers": 1, "art": 12, "treasures": 2, "dug": 4, "commonplaces": 1, "intrinsic": 1, "beauty": 5, "mugs": 2, "snuffers": 1, "prize": 1, "halo": 1, "age": 5, "glowing": 1, "charms": 1, "walls": 10, "ornaments": 3, "household": 2, "pink": 2, "shepherds": 1, "shepherdesses": 1, "gush": 2, "unvalued": 1, "mantel": 1, "eighteenth": 1, "suck": 1, "future": 2, "prized": 1, "trifles": 1, "willow": 1, "pattern": 1, "ranged": 1, "chimneypieces": 1, "2000": 1, "odd": 4, "gold": 6, "rim": 1, "flower": 3, "species": 1, "sarah": 2, "janes": 1, "sheer": 1, "heartedness": 2, "mended": 1, "bracket": 1, "dusted": 2, "china": 5, "bedroom": 3, "furnished": 1, "lodgings": 1, "delicate": 3, "spots": 3, "painfully": 3, "erect": 2, "amiability": 1, "verge": 2, "imbecility": 1, "admire": 3, "considered": 3, "irritates": 2, "jeer": 2, "herself": 8, "admiration": 1, "excuses": 2, "circumstance": 2, "200": 1, "probable": 2, "minus": 1, "cabinet": 1, "depth": 2, "colour": 4, "speculate": 1, "familiar": 2, "loveliness": 2, "2288": 1, "making": 14, "descendants": 1, "lovingly": 1, "artists": 1, "flourished": 1, "sampler": 1, "eldest": 1, "daughter": 2, "spoken": 1, "tapestry": 2, "victorian": 1, "era": 1, "roadside": 1, "hunted": 1, "cracked": 4, "chipped": 1, "weight": 3, "claret": 1, "travellers": 1, "japan": 1, "presents": 2, "ramsgate": 1, "souvenirs": 1, "margate": 2, "escaped": 3, "destruction": 1, "jedo": 1, "ancient": 5, "curios": 1, "howled": 2, "somersault": 1, "lose": 2, "temper": 4, "hulloa": 1, "repeat": 2, "admit": 1, "violence": 1, "coarseness": 1, "especially": 5, "consequence": 5, "tow": 26, "path": 7, "middlesex": 1, "separated": 1, "runs": 6, "charming": 3, "lichen": 1, "creeping": 5, "moss": 2, "growing": 3, "shy": 1, "vine": 1, "peeping": 3, "busy": 3, "sober": 2, "ivy": 2, "clustering": 2, "farther": 1, "fifty": 5, "shades": 4, "tints": 1, "hues": 1, "draw": 6, "paint": 1, "sketch": 1, "ramble": 1, "actual": 1, "dull": 9, "cast": 6, "echo": 1, "rang": 2, "stone": 10, "corridors": 1, "nearer": 6, "creatures": 1, "towns": 3, "cities": 2, "deserted": 1, "sunlightin": 1, "daytime": 1, "alive": 3, "hill": 5, "sides": 2, "lonesome": 2, "frightened": 4, "answering": 2, "throb": 1, "helpless": 3, "rustle": 1, "ghosts": 7, "sighs": 1, "bonfires": 1, "million": 1, "jets": 1, "brave": 1, "studied": 2, "map": 10, "foolishhardly": 1, "charged": 3, "admission": 1, "joke": 3, "quarters": 1, "picked": 5, "absorbed": 2, "persons": 1, "plucked": 1, "courage": 2, "procession": 3, "blessing": 1, "judge": 10, "insisted": 3, "fear": 7, "losing": 1, "largest": 1, "penny": 3, "bun": 1, "swore": 2, "impossible": 5, "expressed": 2, "theory": 4, "whereabouts": 1, "entrance": 3, "beginning": 6, "advisability": 2, "unanimity": 1, "trailed": 3, "pretending": 2, "aiming": 1, "treat": 2, "accident": 6, "consulted": 1, "regular": 3, "length": 9, "infuriated": 1, "mob": 2, "curl": 2, "hair": 10, "extent": 2, "unpopular": 1, "crazy": 2, "sang": 4, "keeper": 7, "climbed": 5, "confused": 4, "whirl": 2, "incapable": 1, "huddled": 4, "wandered": 5, "rushing": 4, "hedge": 2, "reappear": 1, "keepers": 1, "vii": 1, "garb": 1, "chance": 9, "taste": 6, "fashion": 5, "plate": 4, "thomas": 6, "tomb": 7, "graves": 7, "coffins": 1, "skulls": 6, "performs": 2, "tricks": 3, "moulsey": 3, "lock": 62, "boulter": 2, "excepted": 1, "busiest": 1, "tangle": 3, "blazers": 1, "caps": 3, "saucy": 1, "coloured": 3, "parasols": 2, "silken": 1, "cloaks": 1, "streaming": 2, "ribbons": 2, "whites": 1, "quay": 1, "flowers": 3, "hue": 1, "shade": 2, "pell": 1, "mell": 1, "rainbow": 2, "boats": 18, "dotted": 3, "decked": 3, "inhabitants": 3, "costume": 5, "flirt": 1, "jackets": 1, "dresses": 4, "sails": 3, "landscape": 3, "sparkling": 1, "gayest": 1, "sights": 1, "affords": 4, "opportunity": 5, "colours": 2, "natty": 1, "thingsred": 1, "matches": 1, "necktie": 1, "russian": 1, "silk": 1, "waista": 1, "belt": 2, "keeps": 3, "mixtures": 1, "wise": 2, "yellows": 2, "background": 1, "obstinate": 1, "pity": 4, "success": 9, "vexed": 5, "showed": 3, "oriental": 1, "design": 1, "frighten": 1, "respect": 2, "nigger": 1, "huffy": 1, "troubles": 1, "attract": 2, "prettily": 1, "fetching": 1, "tasteful": 1, "worn": 5, "utterly": 3, "excursion": 1, "folk": 7, "misfortune": 1, "lively": 5, "upall": 1, "lace": 3, "silky": 1, "gloves": 1, "photographic": 1, "studio": 1, "costumes": 2, "ridiculous": 3, "seats": 4, "assured": 3, "rubbed": 2, "cushion": 3, "forefinger": 1, "glove": 1, "sighed": 3, "christian": 3, "martyrs": 1, "stake": 1, "occasionally": 2, "splash": 3, "sculling": 11, "stain": 1, "stroke": 14, "feathered": 1, "paused": 3, "blades": 1, "drip": 1, "returning": 2, "smooth": 2, "bow": 16, "accomplished": 3, "oarsman": 1, "flicker": 1, "complain": 1, "lips": 3, "firm": 2, "touched": 2, "visibly": 2, "shrank": 1, "shuddered": 1, "unnerved": 1, "fitful": 2, "rowing": 17, "splashed": 2, "arrangement": 4, "changed": 2, "involuntary": 3, "sigh": 2, "brightened": 2, "thick": 5, "chap": 2, "sensitiveness": 1, "newfoundland": 1, "puppy": 1, "daggers": 1, "rollicking": 1, "dashing": 2, "spray": 1, "fountain": 1, "pint": 2, "offer": 7, "murmur": 3, "covertly": 1, "coats": 7, "protect": 1, "dusty": 3, "tree": 8, "trunks": 2, "brushed": 1, "bolt": 3, "upright": 3, "tripped": 1, "root": 1, "fortunately": 2, "agitated": 1, "grasped": 3, "feared": 1, "rare": 1, "fun": 5, "youri": 1, "sloush": 1, "heartedly": 1, "tuck": 1, "em": 6, "dense": 3, "heno": 1, "funny": 11, "hanker": 1, "tombstones": 1, "village": 15, "churchyard": 4, "recreation": 1, "deny": 1, "chilly": 2, "churches": 1, "wheezy": 2, "epitaphs": 2, "brass": 1, "happiness": 1, "shock": 7, "sextons": 1, "imperturbability": 1, "assume": 2, "inscriptions": 2, "lack": 2, "concealed": 1, "wounds": 4, "leant": 5, "guarded": 1, "smoked": 1, "drank": 2, "gladness": 1, "restful": 4, "scenethe": 1, "porch": 2, "lane": 3, "winding": 4, "tall": 3, "elms": 1, "thatched": 2, "cottages": 2, "hedges": 1, "hollow": 2, "hills": 3, "idyllic": 1, "poetical": 1, "inspired": 1, "sinful": 2, "forgave": 2, "relations": 6, "wickedness": 1, "blessed": 4, "unconscious": 1, "tender": 4, "reverie": 1, "shrill": 1, "piping": 1, "crying": 1, "sur": 6, "bald": 2, "hobbling": 1, "bunch": 1, "keys": 1, "jingled": 1, "motioned": 1, "dignity": 3, "screeching": 1, "lame": 1, "spry": 2, "missis": 1, "repeated": 3, "slay": 1, "tombs": 7, "gritty": 1, "disturb": 1, "chock": 1, "chivying": 1, "tombstone": 3, "nonsense": 4, "bewildered": 2, "yuise": 1, "stranger": 3, "parts": 2, "tombsgravesfolks": 1, "knowcoffins": 1, "untruther": 1, "roused": 2, "tombsnot": 1, "kensal": 1, "cemetery": 1, "grandfather": 1, "vault": 1, "capable": 2, "accommodating": 1, "susan": 1, "brick": 1, "grave": 5, "finchley": 1, "headstone": 1, "bas": 1, "inch": 3, "coping": 1, "burst": 5, "figure": 2, "decipher": 1, "obdurate": 1, "tones": 4, "memorial": 2, "fired": 1, "whispered": 2, "hoarsely": 1, "crypt": 1, "fled": 1, "sped": 1, "calling": 2, "revels": 1, "monumental": 1, "proposedsaid": 1, "shepperton": 3, "lumbering": 1, "barge": 6, "blowed": 1, "continued": 9, "sits": 2, "cheque": 1, "smeared": 1, "effects": 1, "refer": 1, "drawer": 1, "trick": 2, "served": 2, "withdraw": 1, "larking": 1, "everyone": 5, "pumps": 1, "concentrated": 3, "gallon": 2, "jar": 3, "mixing": 2, "cool": 5, "refreshing": 1, "beverage": 1, "slops": 1, "termed": 1, "ginger": 1, "raspberry": 1, "syrup": 1, "dyspepsia": 1, "cause": 2, "steer": 5, "topsy": 1, "turvy": 1, "dived": 1, "holding": 2, "grim": 6, "sticking": 3, "dared": 3, "stay": 5, "madder": 1, "viii": 6, "blackmailing": 2, "pursue": 2, "boorishness": 1, "landowner": 1, "boards": 9, "unchristianlike": 1, "comic": 17, "shameful": 2, "conduct": 4, "information": 1, "buys": 1, "banjo": 10, "willows": 2, "kempton": 1, "park": 7, "lunched": 4, "plateau": 1, "running": 9, "overhung": 1, "commenced": 4, "coursethe": 1, "jamwhen": 1, "shirt": 8, "sleeves": 1, "trespassing": 2, "consideration": 1, "enable": 2, "definite": 1, "hesitation": 1, "required": 6, "assurance": 1, "thanked": 3, "hung": 3, "dissatisfied": 2, "chummy": 1, "disposition": 2, "belonged": 2, "society": 2, "abstain": 1, "declined": 3, "gruffly": 2, "tempted": 1, "bony": 1, "measured": 1, "consult": 1, "chuck": 3, "riverside": 3, "roughs": 2, "income": 1, "slouching": 1, "noodles": 1, "represent": 1, "address": 2, "summon": 2, "damage": 1, "intensely": 4, "lazy": 3, "timid": 2, "imposition": 2, "giving": 6, "exertion": 1, "firmness": 1, "owners": 2, "shown": 1, "selfishness": 1, "riparian": 2, "minor": 1, "tributary": 1, "streams": 2, "backwaters": 3, "chains": 3, "rouses": 1, "evil": 4, "instinct": 3, "tear": 3, "mentioned": 2, "worse": 4, "caused": 2, "slaughter": 1, "burn": 1, "serve": 1, "songs": 7, "instincts": 2, "justice": 2, "degenerate": 1, "vindictiveness": 1, "subject": 10, "promised": 1, "spare": 8, "service": 4, "rendered": 1, "allowed": 3, "implies": 2, "hostess": 2, "beaming": 1, "cheeriness": 1, "generous": 2, "conservatory": 2, "fetch": 3, "smirking": 1, "anticipation": 1, "phrasing": 1, "vocalization": 1, "note": 4, "jerk": 4, "bars": 2, "accompaniment": 3, "easing": 2, "pianist": 10, "verse": 6, "afresh": 1, "repeating": 1, "chorus": 7, "snigger": 2, "blest": 4, "warning": 4, "twell": 1, "addressing": 3, "expectant": 1, "pinafore": 4, "meanyou": 1, "meanthe": 1, "join": 4, "murmurs": 2, "delight": 2, "performance": 3, "prelude": 6, "trial": 5, "jury": 6, "arrives": 1, "takes": 9, "commences": 2, "commencing": 1, "dashes": 1, "lord": 5, "tries": 4, "push": 9, "finds": 2, "stops": 1, "kindly": 5, "indeedgo": 1, "admiral": 2, "argument": 3, "sense": 4, "injustice": 1, "rankling": 1, "requests": 3, "seizing": 3, "considers": 1, "favourable": 1, "opening": 2, "laughter": 5, "compliment": 1, "unequal": 1, "contest": 2, "stronger": 1, "nerved": 1, "explanation": 3, "dawnedlaughing": 1, "jove": 2, "coursei": 1, "jenkins": 1, "appearing": 3, "cellar": 1, "suggesting": 2, "warnings": 1, "approaching": 2, "earthquake": 2, "office": 1, "attorney": 1, "falsetto": 1, "surprise": 8, "audience": 1, "swept": 2, "nono": 3, "cleaned": 3, "polished": 2, "floorno": 1, "dash": 3, "iti": 1, "pardonfunny": 1, "iand": 1, "ioh": 1, "diddle": 12, "de": 1, "ruler": 2, "navee": 2, "chorusit": 1, "dee": 1, "sees": 6, "annoying": 3, "harm": 1, "honestly": 3, "imagines": 1, "parties": 1, "incident": 1, "inner": 2, "recorded": 2, "fashionable": 1, "highly": 2, "cultured": 1, "happyall": 1, "germany": 1, "restless": 3, "uncomfortable": 1, "proceedings": 2, "slow": 3, "truth": 6, "tastes": 1, "played": 8, "morceaux": 1, "masters": 1, "philosophy": 1, "ethics": 1, "flirted": 1, "graceful": 3, "humorousin": 1, "recited": 1, "sentimental": 1, "ballad": 1, "spanish": 1, "weepit": 1, "pathetic": 2, "herr": 9, "slossenn": 7, "boschen": 9, "arrived": 2, "funniest": 2, "written": 2, "emperor": 4, "serious": 4, "reciting": 1, "tragedy": 2, "funnier": 1, "manner": 9, "funnythat": 1, "spoil": 3, "seriousness": 1, "pathos": 2, "irresistibly": 1, "amusing": 3, "fetched": 2, "pleased": 2, "amuse": 1, "unobtrusive": 1, "professor": 3, "accompanied": 1, "suggest": 2, "soulful": 1, "flesh": 5, "murmured": 6, "prepared": 6, "guess": 1, "ignorance": 1, "tittered": 5, "humour": 4, "artful": 1, "progressed": 2, "continuously": 1, "intense": 3, "earnest": 3, "slightest": 2, "annoyance": 3, "indignation": 3, "scowled": 1, "fiercely": 1, "convulsions": 1, "alone": 8, "mock": 1, "seriousnessoh": 1, "surpassed": 1, "glowered": 1, "ferocity": 1, "forewarned": 1, "wailing": 1, "agony": 3, "wept": 1, "amid": 1, "shriek": 2, "popular": 2, "germans": 1, "translate": 1, "effective": 2, "danced": 1, "fists": 1, "insulted": 3, "hartz": 1, "mountains": 2, "lover": 1, "jilted": 1, "spiriti": 1, "details": 1, "sobbed": 2, "acknowledged": 1, "tragic": 1, "situation": 1, "usvery": 1, "unostentatious": 1, "softly": 2, "shady": 4, "servant": 2, "avoiding": 1, "sunbury": 1, "sweetly": 2, "backwater": 7, "attempt": 3, "foot": 3, "crosses": 1, "weirs": 2, "bent": 2, "splendidly": 1, "rhythmical": 1, "swing": 2, "weir": 9, "idiots": 3, "injuring": 2, "violent": 5, "grinding": 3, "sculled": 2, "walton": 4, "tiniest": 1, "abingdon": 6, "corners": 2, "peep": 1, "thanks": 3, "considerate": 1, "fields": 4, "works": 3, "sully": 1, "natured": 1, "ugly": 2, "waltona": 1, "entrenchment": 1, "cromwell": 3, "bradshaw": 1, "charles": 2, "likewise": 2, "sojourned": 1, "scold": 1, "bridle": 1, "curbing": 1, "tongues": 1, "scarce": 1, "winds": 4, "tremendously": 1, "causes": 2, "pulling": 11, "oatlands": 2, "henry": 9, "stole": 2, "grotto": 2, "fee": 1, "duchess": 1, "york": 1, "immense": 3, "special": 2, "graveyard": 1, "epitaph": 1, "inscribed": 1, "thereon": 1, "dare": 2, "deserve": 1, "corway": 1, "stakes": 2, "bend": 5, "bridgewas": 1, "fought": 5, "cassivelaunus": 2, "planting": 1, "choke": 2, "halliford": 1, "longing": 2, "adroit": 1, "movement": 1, "cap": 5, "excitement": 5, "clumsiness": 1, "weybridge": 1, "wey": 1, "navigable": 1, "guildford": 1, "explore": 1, "bourne": 1, "basingstoke": 1, "enter": 1, "closer": 1, "showing": 4, "furious": 2, "barking": 1, "rushed": 8, "fallen": 1, "annoyed": 2, "oilskin": 1, "parcel": 1, "flat": 6, "handle": 3, "rage": 2, "season": 5, "instruction": 1, "ix": 1, "introduced": 1, "heathenish": 1, "ungrateful": 2, "double": 6, "skiff": 5, "towers": 3, "towed": 15, "disappearance": 2, "haste": 2, "speed": 1, "sensation": 4, "missing": 2, "haunted": 3, "saved": 6, "callous": 1, "prone": 1, "consciencenot": 1, "conscienceobject": 1, "worrying": 2, "tired": 6, "unaccountable": 5, "patience": 1, "fold": 1, "revolting": 1, "stretched": 6, "seconds": 4, "twisted": 2, "knots": 1, "ends": 5, "loops": 1, "disentangle": 3, "honourable": 1, "exceptions": 1, "professionconscientious": 1, "linestow": 1, "crochet": 1, "knit": 1, "antimacassars": 1, "sincerely": 1, "careless": 4, "looped": 1, "cautiously": 2, "laid": 7, "gently": 6, "lifted": 1, "scientifically": 1, "unravel": 1, "swaddling": 1, "infant": 1, "unwound": 1, "mat": 1, "connection": 4, "thinks": 7, "fault": 5, "fishing": 14, "net": 1, "properly": 4, "dummy": 2, "grunts": 1, "wildly": 4, "lays": 2, "wound": 2, "muddle": 1, "rests": 2, "unwind": 1, "exclaims": 1, "scaffolding": 1, "pole": 12, "entangled": 2, "dances": 3, "rope": 2, "hauling": 1, "tighter": 1, "climbs": 1, "drifted": 4, "knowledge": 3, "boveney": 2, "helplessly": 2, "witnessed": 3, "countenance": 1, "indignant": 4, "act": 1, "truant": 1, "incidents": 1, "briskly": 1, "animated": 1, "vainly": 1, "shrieking": 1, "frantic": 4, "distress": 1, "scull": 8, "hook": 9, "dropped": 5, "floating": 4, "rapidly": 2, "calls": 2, "politely": 1, "shouts": 4, "tomdick": 1, "affably": 1, "confound": 2, "dunder": 1, "springs": 1, "roars": 1, "curses": 5, "pitch": 1, "remembering": 1, "frequent": 1, "chattering": 1, "offering": 1, "resistance": 1, "reminding": 1, "example": 3, "oblivious": 1, "discussing": 1, "adrift": 1, "maidenhead": 6, "cookham": 4, "absorbing": 1, "fate": 4, "overtaken": 1, "disturbed": 3, "necessary": 3, "flashed": 3, "hitcher": 6, "tidied": 2, "hulking": 1, "chaps": 1, "sadness": 6, "glance": 7, "restraining": 1, "maiden": 2, "recover": 4, "clasped": 1, "auntie": 1, "sympathy": 2, "tower": 1, "shelves": 1, "noticing": 2, "pace": 5, "scattered": 2, "reposeful": 1, "attitudes": 3, "precise": 4, "noise": 4, "ripping": 1, "linen": 1, "sheets": 1, "larboard": 1, "reclined": 1, "disembarked": 1, "starboard": 1, "hooks": 1, "carpet": 1, "lighten": 2, "shouting": 3, "urging": 1, "gallop": 1, "realised": 1, "occupied": 2, "distance": 7, "hid": 3, "sorry": 8, "mishap": 1, "fashionand": 1, "docould": 1, "similar": 1, "misfortunes": 1, "besides": 5, "risk": 1, "theirs": 1, "hitched": 1, "overturns": 1, "cuts": 1, "plan": 5, "butt": 3, "experiences": 2, "giggles": 1, "undo": 1, "twist": 1, "necks": 3, "strangled": 1, "breathless": 3, "drifts": 1, "turns": 3, "pinning": 1, "frock": 2, "occurs": 1, "pin": 1, "ease": 3, "aground": 2, "jump": 3, "stopgo": 1, "ongo": 1, "emily": 6, "asks": 1, "knowdon": 1, "stopping": 5, "wayyou": 1, "nicely": 2, "shawl": 2, "hers": 7, "mary": 2, "cow": 2, "chivy": 1, "penton": 2, "important": 1, "staines": 7, "shutting": 1, "runnymead": 1, "afterward": 1, "weary": 4, "trudged": 2, "least": 12, "seriously": 2, "figurative": 1, "ladycousin": 1, "sideand": 1, "goring": 10, "inat": 1, "benson": 2, "dusk": 1, "lockwallingfordand": 1, "cleeve": 2, "youyou": 2, "hesitatingly": 1, "offend": 2, "twilight": 5, "ghost": 2, "drowned": 6, "judgment": 2, "excessive": 1, "punishment": 1, "reassure": 1, "fast": 2, "wallingford": 13, "clearly": 2, "marked": 1, "reliable": 1, "recollected": 2, "dreaming": 3, "gathering": 1, "hobgoblins": 1, "banshees": 1, "wisps": 1, "lure": 2, "pools": 1, "hymns": 1, "reflections": 1, "strains": 3, "concertina": 2, "thenfar": 1, "orpheus": 1, "lute": 1, "apollo": 1, "sounded": 2, "heavenly": 3, "melody": 3, "harrowed": 1, "harmony": 2, "correctly": 1, "performed": 2, "jerked": 4, "spasmodically": 1, "variations": 1, "accordion": 1, "reassuring": 1, "alongside": 1, "contained": 3, "provincial": 1, "arrys": 1, "arriets": 1, "sail": 18, "attractive": 1, "lovable": 2, "lor": 2, "tight": 3, "bless": 4, "sounding": 1, "gratitude": 2, "soldiers": 2, "faust": 1, "x": 1, "appeal": 3, "contrariness": 1, "overcome": 2, "virtuous": 1, "comfortably": 4, "appointed": 2, "drained": 2, "island": 14, "pacific": 1, "father": 16, "dragging": 1, "tons": 1, "originally": 1, "magna": 3, "charta": 3, "soft": 5, "valley": 2, "inlets": 1, "tiny": 3, "earlier": 1, "coal": 2, "satisfied": 2, "calledand": 1, "elm": 1, "spreading": 1, "roots": 1, "fastened": 4, "dispensed": 1, "bargained": 1, "abstract": 1, "arches": 3, "croquet": 1, "fitted": 2, "estimate": 1, "sockets": 4, "tale": 3, "demons": 2, "kick": 1, "struggled": 2, "drown": 3, "hinges": 2, "nipped": 1, "wrestling": 1, "hoop": 1, "endeavouring": 2, "persuade": 2, "covering": 1, "unrolled": 2, "receive": 3, "bungled": 1, "process": 2, "superhuman": 1, "effort": 7, "tucked": 1, "freedomthe": 1, "birthright": 1, "englishman": 1, "violently": 2, "considerably": 1, "interfere": 1, "guessed": 2, "troublesome": 1, "involved": 1, "wriggling": 1, "spoke": 10, "cuckoo": 2, "mummy": 1, "suffocated": 1, "withstand": 1, "undid": 1, "cleared": 1, "decks": 1, "boil": 3, "sputtering": 1, "loudly": 1, "overhear": 1, "insteadtea": 1, "boils": 1, "trickery": 1, "lantern": 2, "squatted": 3, "breadth": 1, "clank": 1, "cutlery": 1, "crockery": 1, "sets": 1, "molars": 1, "contentment": 2, "exhibited": 2, "bumped": 1, "fullhow": 1, "conscience": 1, "obtained": 2, "forgiving": 2, "substantial": 2, "digested": 1, "mealso": 1, "domination": 1, "intellect": 3, "digestive": 1, "organs": 1, "unless": 1, "wills": 1, "dictates": 1, "emotions": 1, "passions": 1, "spoonsful": 1, "brain": 1, "strength": 5, "quivering": 1, "soar": 1, "whirling": 1, "lanes": 1, "flaming": 1, "eternity": 1, "muffins": 1, "soulless": 2, "beast": 1, "fielda": 1, "brainless": 1, "animal": 3, "listless": 1, "unlit": 1, "ray": 2, "grin": 2, "laughdrivel": 1, "folly": 1, "splutter": 3, "ninny": 1, "wit": 2, "kittens": 1, "alcohol": 1, "veriest": 2, "sorriest": 1, "slaves": 1, "morality": 1, "righteousness": 1, "vigilantly": 1, "reign": 2, "unsought": 1, "citizen": 1, "fathera": 1, "snappy": 1, "tempered": 2, "beamed": 2, "corn": 1, "wishes": 1, "desires": 2, "shudder": 1, "ware": 1, "wheat": 1, "observing": 1, "unpleasant": 2, "treading": 1, "sized": 2, "advising": 1, "thisaway": 1, "temptation": 1, "leading": 6, "longed": 2, "possibility": 1, "handy": 3, "islands": 8, "drains": 1, "wales": 1, "slightly": 1, "lurched": 1, "undress": 3, "grope": 1, "separate": 2, "itone": 1, "crawling": 1, "compass": 1, "lying": 10, "pillow": 5, "joe": 5, "valiantly": 1, "bumps": 2, "doleful": 1, "pig": 1, "whistle": 5, "bangs": 1, "novelty": 1, "hardness": 1, "cramped": 1, "branches": 3, "nightfor": 1, "disappeared": 2, "morningkept": 1, "digging": 1, "spine": 1, "swallowed": 3, "gimlet": 2, "unkind": 2, "owe": 1, "month": 6, "otherwise": 1, "accumulate": 2, "excruciating": 1, "wrench": 1, "ached": 1, "aboutsome": 1, "crept": 4, "sisterconversing": 1, "mighty": 2, "mysteries": 1, "vast": 2, "childish": 1, "ears": 2, "awe": 2, "strayed": 2, "temple": 2, "taught": 2, "worship": 1, "echoing": 1, "dome": 1, "spans": 1, "vista": 1, "shadowy": 3, "hoping": 1, "hovering": 1, "sorrows": 1, "fevered": 1, "stained": 2, "smiles": 1, "flushed": 1, "cheek": 2, "pain": 7, "moan": 1, "mightier": 1, "wondrous": 3, "sorrow": 3, "angels": 2, "rode": 3, "goodly": 1, "knights": 5, "tangled": 2, "briars": 1, "gloom": 3, "knight": 4, "comrades": 2, "sorely": 1, "grieving": 1, "mourning": 1, "castle": 4, "journeying": 1, "stayed": 2, "logs": 1, "burned": 2, "hall": 7, "comrade": 1, "ragged": 1, "beggar": 1, "shone": 1, "radiance": 1, "joy": 5, "questioned": 2, "befallen": 1, "torn": 1, "bleeding": 2, "nigh": 1, "unto": 4, "lo": 2, "devious": 1, "paths": 3, "darkness": 4, "entranced": 1, "whereof": 2, "faded": 1, "kneeling": 1, "saint": 1, "forest": 1, "xi": 1, "heroism": 1, "determination": 2, "moral": 1, "historical": 1, "retrospect": 1, "inserted": 1, "schools": 1, "awake": 3, "watches": 1, "earthly": 4, "utter": 2, "absurdity": 1, "gippings": 1, "occurrence": 1, "shortest": 1, "bargain": 2, "hauled": 2, "ministers": 1, "grace": 4, "defend": 1, "shame": 3, "flung": 2, "sprang": 2, "washed": 7, "shaved": 1, "received": 4, "g": 4, "unbolted": 2, "anathematized": 1, "decent": 1, "unlocked": 1, "shops": 3, "foggy": 2, "holborn": 1, "shutter": 1, "bus": 1, "policeman": 4, "cart": 1, "cabbages": 2, "dilapidated": 1, "counted": 2, "stooped": 1, "eyeing": 1, "suspicion": 4, "listened": 3, "neighbouring": 1, "obliged": 2, "constable": 1, "guardian": 1, "severely": 1, "advice": 2, "redressing": 1, "wakeful": 1, "chess": 3, "game": 2, "enliven": 1, "horribly": 1, "dismal": 4, "policemen": 2, "undisguised": 1, "lanterns": 1, "slinking": 1, "hiding": 2, "doorways": 1, "regulation": 1, "flip": 1, "flop": 1, "distrustful": 1, "rout": 1, "stroll": 3, "constables": 1, "key": 1, "scuttleful": 1, "coals": 1, "dropping": 1, "burglars": 1, "police": 4, "detectives": 1, "handcuff": 1, "morbidly": 1, "pictured": 1, "believing": 1, "sentenced": 1, "penal": 1, "servitude": 1, "overcoat": 1, "true": 4, "finishing": 1, "prod": 1, "aid": 1, "sending": 1, "poked": 1, "shivered": 1, "overnight": 1, "fling": 1, "shawls": 1, "joyous": 1, "delicious": 1, "tempting": 1, "precedence": 1, "retiring": 3, "vent": 1, "howl": 4, "horrors": 1, "sorted": 1, "relish": 2, "snags": 1, "weeds": 1, "wormed": 1, "branch": 3, "dipped": 1, "knife": 6, "blowing": 3, "surface": 1, "pluck": 4, "spluttered": 2, "duffers": 1, "missed": 2, "worlds": 7, "accidentally": 1, "awfully": 1, "drivelling": 1, "imbecile": 1, "louder": 3, "mistaken": 3, "whereupon": 2, "roaring": 2, "amused": 3, "ar": 1, "n": 1, "youyougoing": 1, "shrieks": 2, "peals": 1, "shirtit": 1, "springing": 1, "careful": 3, "deuce": 1, "scrambled": 5, "picnics": 1, "yachts": 1, "pined": 1, "mouths": 2, "eggsor": 1, "preventing": 2, "chivied": 2, "fork": 2, "harassing": 1, "flicking": 1, "fingers": 1, "performing": 1, "feat": 1, "culinary": 1, "indian": 1, "sandwich": 1, "dish": 3, "incantations": 1, "scalded": 1, "dancing": 2, "operations": 2, "anticipated": 1, "teaspoonful": 1, "unappetizing": 1, "aids": 1, "housekeeping": 1, "remind": 1, "june": 2, "1215": 1, "drawn": 6, "yeomen": 2, "sons": 1, "homespun": 1, "cloth": 2, "dirk": 1, "witness": 1, "writing": 1, "stupendous": 1, "page": 1, "meaning": 2, "translated": 1, "oliver": 1, "morningsunny": 1, "thrill": 1, "stir": 2, "john": 14, "duncroft": 2, "echoed": 1, "clang": 1, "armed": 2, "clatter": 2, "horses": 1, "captains": 1, "surly": 1, "jests": 1, "bearded": 1, "bowmen": 1, "billmen": 1, "pikemen": 1, "foreign": 1, "spearmen": 1, "companies": 1, "squires": 2, "ridden": 1, "travel": 1, "townsmen": 1, "doors": 3, "groups": 3, "woe": 1, "betide": 1, "sword": 2, "plaintiff": 1, "executioner": 1, "tempestuous": 1, "pays": 1, "sparing": 1, "pleases": 1, "barons": 8, "troops": 3, "bellow": 1, "roystering": 1, "gamble": 1, "quarrel": 1, "deepens": 1, "firelight": 1, "sheds": 1, "uncouth": 1, "forms": 2, "wondering": 5, "brawny": 1, "wenches": 1, "bandy": 1, "ale": 4, "jest": 1, "jibe": 1, "swaggering": 1, "troopers": 1, "unlike": 1, "swains": 1, "despised": 1, "apart": 1, "vacant": 2, "grins": 1, "broad": 1, "peering": 2, "glitter": 1, "lights": 2, "camps": 1, "followers": 1, "mustered": 1, "mercenaries": 3, "hover": 1, "crouching": 2, "wolves": 1, "sentinel": 1, "twinkling": 1, "fires": 1, "height": 2, "thame": 2, "ages": 1, "workmen": 1, "pavilion": 1, "yester": 1, "eve": 1, "carpenters": 1, "nailing": 1, "tiers": 1, "prentices": 1, "stuffs": 1, "guttural": 1, "bass": 2, "score": 2, "stalwart": 1, "halbert": 1, "menbarons": 1, "theseand": 1, "halt": 1, "bands": 1, "casques": 1, "breastplates": 1, "steeds": 1, "horsemen": 1, "galloping": 1, "group": 3, "banners": 1, "fluttering": 1, "lazily": 2, "breeze": 4, "ranks": 3, "baron": 1, "war": 2, "passes": 5, "serfs": 1, "vassals": 1, "slope": 1, "cooper": 1, "rustics": 1, "townsfolk": 1, "bustle": 2, "version": 1, "event": 1, "shake": 4, "coracleswhich": 1, "favour": 3, "poorer": 1, "rapids": 1, "sturdy": 1, "rowers": 2, "crowding": 1, "bear": 5, "fateful": 1, "charter": 2, "waits": 1, "signing": 2, "noon": 1, "patient": 3, "slippery": 2, "stolen": 1, "heels": 1, "charters": 1, "liberty": 4, "grip": 2, "slid": 2, "wriggled": 1, "vain": 2, "dust": 3, "risen": 1, "larger": 2, "pattering": 1, "hoofs": 1, "pushes": 1, "cavalcade": 1, "lords": 1, "flank": 1, "ride": 2, "midst": 3, "rides": 1, "greets": 1, "honeyed": 1, "dismount": 1, "casts": 1, "hurried": 3, "hem": 1, "fierce": 2, "unsuspecting": 1, "horseman": 1, "desperate": 3, "unready": 1, "rebellious": 1, "rue": 1, "thwart": 1, "bolder": 1, "richard": 1, "sinks": 1, "drops": 2, "rein": 1, "dismounts": 1, "foremost": 1, "mailed": 1, "hilt": 1, "runningmede": 2, "swift": 1, "current": 4, "ponderous": 1, "grumble": 1, "grate": 1, "cleaves": 1, "cornerstone": 1, "xii": 1, "anne": 6, "boleyn": 2, "disadvantages": 1, "nation": 1, "search": 1, "homeless": 1, "houseless": 1, "prepares": 1, "mustard": 10, "sailing": 7, "fishers": 2, "cursed": 2, "conjuring": 1, "remarked": 1, "rested": 3, "recalled": 1, "prosaic": 1, "tuft": 1, "polishing": 1, "stands": 6, "cottage": 3, "signed": 2, "commit": 1, "personal": 2, "customer": 1, "priory": 2, "grounds": 3, "ankerwyke": 1, "hever": 2, "kent": 4, "albans": 3, "spooning": 1, "courting": 2, "edward": 4, "thrall": 1, "photographs": 1, "relatives": 1, "pausing": 1, "coldly": 1, "papa": 1, "items": 1, "news": 1, "opinions": 1, "closes": 1, "shuts": 1, "relied": 1, "civilised": 1, "community": 1, "poke": 1, "uninteresting": 5, "garden": 3, "buckinghamshire": 1, "unexpectedly": 1, "mooning": 1, "wraysbury": 1, "blushed": 1, "billing": 1, "cooing": 1, "drat": 2, "albansnice": 1, "kissing": 1, "abbey": 13, "pirates": 2, "marriage": 1, "delightful": 5, "bells": 2, "ouseley": 1, "drunkso": 1, "confessor": 1, "earl": 4, "godwin": 2, "proved": 1, "guilty": 3, "encompassed": 1, "broke": 7, "choked": 1, "nearing": 1, "stretches": 2, "albert": 1, "victoria": 1, "datchet": 4, "august": 2, "diggings": 1, "clematis": 1, "creeper": 1, "honeysuckle": 4, "honey": 1, "suckle": 1, "wore": 4, "goodish": 2, "direct": 2, "stag": 4, "itno": 1, "manor": 4, "theredidn": 1, "thereharris": 1, "informant": 1, "greatest": 3, "realise": 2, "ideals": 1, "hollowness": 1, "traps": 1, "landlord": 7, "traveller": 1, "occasion": 1, "billiard": 3, "needn": 1, "sensibly": 1, "greeting": 1, "fourteenth": 1, "meek": 2, "suggestions": 2, "stables": 1, "cellars": 1, "scorn": 2, "nooks": 2, "roughing": 1, "itshe": 1, "recommend": 3, "mindbut": 1, "beershop": 2, "eton": 1, "panting": 1, "bargeman": 1, "rooms": 3, "enlivened": 1, "pains": 1, "27": 2, "requested": 2, "disguise": 2, "assumed": 1, "producing": 1, "peculiarly": 2, "unattractive": 1, "suggestive": 2, "messenger": 1, "occupants": 2, "paralysed": 1, "pigstye": 1, "disused": 1, "limekiln": 1, "placeat": 1, "powered": 1, "emotion": 3, "unable": 5, "sustain": 1, "letting": 2, "fainted": 1, "seize": 1, "consciousness": 1, "roomed": 1, "mothergood": 1, "allfive": 1, "poundsand": 1, "pots": 1, "2ft": 1, "6in": 1, "truckle": 1, "bare": 2, "bathed": 1, "tugged": 1, "monkey": 2, "tackled": 1, "seldom": 1, "universe": 1, "spoonful": 1, "reckless": 1, "daresay": 2, "extravagant": 1, "offers": 1, "absurdly": 1, "proportion": 2, "value": 1, "mountain": 1, "switzerland": 1, "shanty": 1, "kicked": 2, "francs": 1, "scandalous": 1, "childhood": 1, "tin": 11, "pine": 2, "juice": 2, "spoon": 3, "opener": 1, "scissors": 2, "spiky": 1, "uninjured": 1, "teacup": 1, "poised": 1, "stretchers": 2, "dangers": 1, "brings": 1, "shows": 3, "stirring": 2, "anew": 1, "exaggerations": 1, "hammered": 1, "flattened": 1, "battered": 2, "geometrybut": 1, "unearthly": 1, "hideousness": 1, "dent": 1, "mocking": 3, "sank": 2, "hurled": 2, "rowed": 3, "snobby": 1, "haunt": 1, "overdressed": 1, "showy": 1, "chiefly": 1, "dudes": 1, "ballet": 2, "witch": 1, "riversteam": 1, "launches": 6, "journal": 1, "duke": 1, "heroine": 3, "volume": 1, "novel": 3, "dines": 1, "spree": 1, "leisurely": 1, "locks": 7, "clieveden": 1, "blended": 1, "fairy": 3, "unbroken": 1, "sweetest": 3, "lingeringly": 1, "peace": 6, "stiffish": 1, "sprung": 1, "upin": 1, "veers": 1, "consistently": 1, "ways": 1, "probation": 1, "sparks": 1, "fly": 4, "upward": 1, "bellied": 1, "grumbled": 2, "steered": 1, "thrilling": 1, "yetexcept": 1, "dreams": 2, "bearing": 3, "onward": 1, "plodding": 1, "puny": 1, "clay": 4, "tortuously": 1, "throbbing": 1, "raising": 1, "limbs": 1, "brothers": 1, "punt": 12, "moored": 3, "fishermen": 4, "skimmed": 1, "chairs": 3, "intently": 1, "mystic": 2, "tinged": 1, "towering": 1, "glory": 2, "enchantment": 1, "ecstatic": 1, "purple": 1, "sky": 2, "gloaming": 1, "wrapping": 2, "legend": 1, "lake": 3, "realm": 2, "beings": 1, "sorting": 1, "usnot": 1, "cursory": 2, "comprehensive": 1, "embraced": 1, "career": 1, "included": 1, "connected": 1, "usgood": 1, "grateful": 2, "shocked": 2, "grieved": 2, "boatsbetter": 1, "xiii": 1, "bisham": 7, "medmenham": 4, "monks": 4, "decides": 1, "civil": 1, "departure": 2, "imposing": 3, "launch": 20, "receipts": 1, "hindering": 1, "pleasantest": 1, "centres": 1, "bustling": 1, "neverthelessstanding": 1, "shattered": 2, "travels": 1, "owned": 1, "algar": 1, "conquering": 1, "matilda": 1, "earls": 1, "warwick": 2, "paget": 1, "councillor": 1, "successive": 1, "sovereigns": 1, "quarry": 2, "meadows": 2, "narrow": 2, "climbing": 1, "glades": 1, "scented": 1, "memories": 2, "vistas": 1, "sonning": 6, "fairer": 1, "rung": 1, "templars": 2, "cleves": 1, "melodramatic": 1, "properties": 1, "contains": 2, "chamber": 2, "secret": 2, "holy": 1, "walks": 1, "trivial": 1, "kingdoms": 1, "salisbury": 1, "poitiers": 1, "inspecting": 1, "monuments": 1, "beeches": 1, "shelley": 1, "west": 1, "composed": 1, "revolt": 1, "islam": 1, "hurley": 2, "dating": 3, "quote": 1, "phraseology": 1, "sebert": 1, "offa": 1, "danes": 4, "invading": 1, "encamped": 1, "gloucestershire": 1, "nestling": 3, "hell": 1, "club": 2, "commonly": 1, "notorious": 1, "wilkes": 1, "member": 2, "fraternity": 3, "motto": 1, "invitation": 1, "doorway": 1, "bogus": 1, "congregation": 1, "irreverent": 1, "jesters": 1, "founded": 3, "monastery": 1, "sterner": 1, "type": 3, "revellers": 1, "cistercian": 1, "thirteenth": 1, "tunics": 1, "cowls": 1, "mass": 1, "prayer": 2, "themthe": 1, "whisperings": 1, "windshould": 1, "truer": 1, "heaven": 4, "myriad": 1, "hambledon": 2, "greenlands": 1, "residence": 1, "newsagenta": 1, "unassuming": 1, "regions": 1, "vigorous": 2, "chatting": 3, "genially": 1, "throughuntil": 1, "henley": 9, "tolerably": 1, "cats": 5, "pussy": 1, "tickle": 1, "sticks": 1, "rigid": 1, "wipes": 1, "gentleness": 1, "meets": 1, "contenting": 1, "clouting": 1, "terriers": 2, "christians": 1, "appreciable": 2, "reformation": 1, "rowdiness": 1, "lobby": 4, "haymarket": 2, "shopping": 1, "mastiff": 1, "collies": 1, "bernard": 1, "retrievers": 1, "newfoundlands": 1, "hound": 1, "poodle": 6, "mangy": 1, "bull": 7, "lowther": 1, "arcade": 1, "animals": 1, "yorkshire": 2, "tykes": 1, "peacefulness": 1, "calmness": 1, "resignationof": 1, "gentle": 4, "pervaded": 1, "chained": 1, "judging": 2, "dignified": 3, "dreamlessly": 1, "haughty": 1, "shadow": 1, "provocation": 1, "fore": 1, "yelp": 2, "experiment": 2, "satisfactory": 4, "vigorously": 1, "attacked": 1, "collie": 2, "foxey": 1, "curiously": 2, "impartial": 1, "uninterrupted": 2, "equally": 2, "willing": 1, "tyke": 2, "canine": 1, "hearths": 1, "homes": 1, "depended": 1, "fray": 1, "indiscriminately": 1, "biting": 1, "pandemonium": 1, "terrific": 1, "vestry": 1, "murdered": 1, "poles": 1, "riot": 1, "lamb": 1, "kissed": 2, "nasty": 1, "brutes": 1, "darted": 1, "trot": 2, "joythe": 1, "warrior": 1, "enemy": 2, "handsthe": 1, "uttered": 1, "scots": 1, "hilland": 1, "prey": 2, "sinewy": 1, "updid": 1, "trotted": 1, "assassin": 2, "inquiring": 1, "chilled": 1, "boldest": 1, "abruptly": 1, "follows": 2, "nonot": 1, "allcertainlydon": 1, "allquite": 1, "thanksnot": 1, "allvery": 1, "fitting": 1, "groove": 1, "unimportant": 1, "shrink": 1, "piteously": 1, "marketing": 1, "revictualled": 1, "vegetablesthat": 1, "unhealthy": 2, "vegetables": 2, "potatoes": 7, "bushel": 1, "peas": 4, "gooseberry": 1, "tarts": 1, "mutton": 1, "foraged": 1, "successes": 1, "impressive": 1, "ostentatious": 1, "basket": 5, "adopting": 2, "principle": 2, "collection": 2, "baskets": 2, "final": 3, "spectacle": 1, "curs": 1, "bulged": 1, "lime": 1, "baker": 1, "confectioner": 1, "haired": 1, "cheesemonger": 1, "fruiterer": 1, "stray": 1, "boatman": 7, "informing": 1, "numbers": 1, "houseboats": 1, "hate": 2, "strangle": 1, "blatant": 1, "bumptiousness": 1, "knack": 1, "rousing": 1, "yearn": 2, "hatchet": 1, "arrows": 1, "breach": 1, "lordly": 1, "ensure": 1, "verdict": 1, "justifiable": 1, "homicide": 1, "boastful": 1, "delay": 1, "aggravation": 1, "sighting": 1, "drift": 2, "whistling": 2, "anecdote": 1, "boiler": 2, "reverse": 1, "engines": 1, "narrative": 1, "instruct": 1, "rightyou": 1, "oneleave": 1, "younow": 1, "assistance": 2, "thank": 2, "aristocratic": 1, "beanfeast": 1, "messrs": 1, "cubit": 1, "bermondsey": 1, "saucepan": 1, "accustomed": 2, "windsora": 1, "mechanical": 1, "monstrositieswith": 1, "containing": 2, "description": 1, "glimpse": 1, "owed": 2, "families": 1, "hardy": 1, "spokesman": 1, "winning": 2, "wherewhere": 1, "stolid": 3, "brand": 1, "pump": 1, "chancing": 1, "germs": 1, "poison": 1, "boiling": 1, "settling": 1, "westward": 1, "gaze": 4, "sluggish": 2, "quietest": 1, "peacefullest": 1, "contentedmore": 1, "bodied": 1, "developed": 1, "serene": 1, "abreast": 1, "cosily": 1, "emptied": 2, "escaping": 2, "wargrave": 6, "marsh": 1, "saving": 1, "studded": 1, "surrounded": 1, "menacing": 1, "torture": 1, "imprisonment": 1, "dares": 1, "watersi": 1, "boors": 1, "claim": 1, "threaten": 1, "itbut": 1, "skill": 4, "avoid": 5, "meadow": 2, "feed": 1, "gravy": 3, "hundreds": 1, "tumbled": 1, "climb": 2, "descending": 1, "practicable": 1, "froze": 1, "veins": 1, "headand": 1, "headsticking": 1, "deadand": 1, "heredarn": 1, "rescuing": 1, "pievery": 1, "damaged": 2, "harristumbled": 1, "grubby": 1, "gully": 1, "conjecture": 1, "believes": 1, "planned": 1, "unjust": 1, "poet": 2, "calumny": 1, "xiv": 1, "waxworks": 1, "stew": 9, "sarcastic": 2, "studies": 1, "discouragement": 2, "difficulties": 1, "amateur": 2, "learning": 3, "play": 9, "bagpipes": 6, "strangeness": 2, "swans": 9, "troubled": 1, "shiplake": 4, "mellowed": 1, "bends": 1, "lingers": 1, "retina": 1, "dragon": 1, "boasts": 1, "leslie": 2, "r": 1, "hodgson": 2, "ilk": 1, "depicted": 1, "imagined": 2, "enjoying": 3, "author": 1, "andmore": 1, "stillwas": 1, "bequeathed": 1, "annually": 1, "divided": 1, "easter": 1, "undutiful": 1, "untruths": 1, "rumoured": 1, "thingsor": 1, "themand": 1, "wax": 1, "tennyson": 1, "placid": 1, "hushed": 2, "rustic": 1, "arry": 1, "fitznoodle": 1, "vanished": 1, "mortar": 1, "roses": 1, "bursting": 1, "splendour": 1, "veritable": 1, "courtyard": 1, "gossip": 2, "politics": 1, "awkward": 3, "passages": 2, "roamed": 1, "odds": 2, "fascinating": 2, "peel": 3, "peeling": 3, "undertaking": 1, "cheerfully": 1, "skittishly": 1, "potato": 4, "peeled": 1, "leftat": 1, "itit": 1, "pea": 1, "nut": 1, "scraped": 1, "potatoesall": 1, "warts": 1, "hollows": 1, "require": 2, "scraping": 2, "scrapings": 1, "economy": 1, "cabbage": 1, "peck": 1, "overhauled": 2, "remnants": 1, "pork": 1, "potted": 1, "salmon": 1, "advantage": 1, "fished": 2, "thicken": 1, "ingredients": 1, "evinced": 1, "strolled": 1, "reappearing": 1, "contribution": 1, "genuine": 1, "assist": 2, "helped": 2, "precedent": 2, "safe": 2, "experiments": 1, "progress": 1, "piquant": 1, "palate": 1, "hackneyed": 3, "nourishing": 1, "softer": 1, "poema": 1, "nutritious": 2, "cherry": 1, "manifested": 1, "challenge": 1, "threatening": 1, "attitude": 3, "spit": 1, "growled": 1, "teach": 3, "nosed": 1, "scoundrel": 1, "spout": 1, "constitutional": 2, "mud": 7, "mixture": 1, "growl": 1, "rapid": 1, "goodsaid": 1, "soothed": 1, "nerves": 1, "twanged": 1, "evenings": 1, "unnerve": 1, "howling": 2, "retort": 1, "postpone": 1, "sorryfor": 1, "himbut": 1, "injure": 2, "practising": 1, "captured": 1, "evidence": 1, "bound": 1, "efforts": 4, "elapsed": 1, "coldnessthe": 1, "despaired": 1, "advertised": 1, "sale": 2, "sacrifice": 1, "card": 1, "disheartening": 1, "acquire": 2, "studying": 1, "amount": 3, "contend": 1, "members": 2, "active": 2, "unfeelingly": 1, "practise": 2, "committed": 1, "jefferson": 3, "describe": 2, "mercy": 1, "gurgle": 1, "successful": 1, "precautions": 1, "affect": 1, "shark": 1, "guineawhere": 1, "visitor": 2, "caution": 2, "earshot": 1, "confessed": 1, "listening": 2, "perform": 1, "tune": 6, "startat": 1, "magnificently": 1, "collapsed": 1, "hiss": 1, "complaints": 1, "insufficiency": 1, "repertoirenone": 1, "campbells": 1, "hoorayhooray": 1, "scotland": 1, "strangers": 4, "guesses": 1, "disagreeable": 1, "grunted": 2, "regatta": 1, "company": 3, "homeas": 1, "coldish": 1, "conjured": 1, "shapeless": 1, "giant": 1, "glow": 1, "worm": 2, "snug": 1, "pecking": 1, "chunks": 1, "cheery": 3, "knives": 1, "filling": 1, "space": 2, "overflowing": 1, "prior": 1, "uncertainties": 1, "skiplake": 2, "striking": 1, "thoughtfully": 2, "response": 1, "hopefully": 1, "hallooed": 1, "becoming": 3, "crammed": 1, "knocking": 1, "cottagers": 1, "householders": 1, "apartments": 1, "assaulting": 1, "hits": 1, "refuses": 1, "overdo": 2, "despairingly": 1, "skin": 1, "babes": 1, "hopeyes": 1, "novels": 1, "resolved": 1, "strictly": 1, "truthful": 1, "employ": 1, "phrases": 1, "glimmer": 1, "flickering": 1, "thenoh": 1, "divinest": 1, "bark": 2, "sleepersi": 1, "sleepers": 1, "oneand": 1, "blackness": 1, "tiredness": 1, "screaming": 1, "safely": 1, "swan": 4, "nest": 1, "defeated": 2, "defended": 1, "paddled": 1, "sleepily": 1, "count": 4, "facts": 1, "trials": 1, "toddy": 3, "examined": 1, "recollection": 3, "wandering": 1, "routed": 2, "hazy": 1, "remembrance": 1, "hearing": 1, "muttering": 1, "xv": 1, "duties": 1, "tells": 1, "scepticism": 1, "generation": 3, "recollections": 1, "rafting": 3, "beginner": 1, "punting": 6, "friendship": 1, "housework": 1, "partook": 2, "non": 1, "dainties": 1, "continual": 1, "afford": 1, "insight": 1, "posed": 1, "menamely": 1, "manages": 1, "chime": 1, "share": 2, "fascinates": 1, "breaks": 1, "passion": 1, "wing": 1, "possession": 1, "preservation": 1, "crave": 1, "itat": 2, "appears": 4, "worries": 2, "scrupulous": 1, "due": 1, "crew": 1, "ridiculed": 1, "hegeorge": 1, "himselfwho": 1, "skulks": 1, "hadmost": 1, "fully": 1, "compelled": 1, "support": 1, "rejoined": 1, "passenger": 1, "superintended": 1, "slaved": 1, "difficulty": 3, "arranging": 1, "youngsters": 3, "similarly": 1, "cushions": 1, "encourages": 1, "marvellous": 1, "feats": 2, "drawls": 1, "whiffs": 1, "perspiring": 1, "novices": 3, "biffles": 1, "jack": 9, "afternoonnever": 1, "partially": 1, "wakes": 1, "appealed": 2, "recollects": 1, "remembers": 1, "unusually": 1, "waylikewise": 1, "adds": 2, "speaker": 1, "reaching": 2, "exaggerate": 2, "reprovingly": 1, "exhausted": 1, "conversational": 1, "proud": 2, "oarsmen": 1, "elders": 1, "digest": 1, "faith": 1, "wegeorge": 1, "myselftook": 1, "un": 1, "plied": 1, "customary": 1, "onesthe": 1, "honoured": 1, "pastand": 1, "invented": 1, "episode": 1, "degree": 1, "oursa": 1, "believed": 2, "mocked": 1, "recounting": 1, "stories": 2, "oarsmanship": 1, "contributing": 1, "threepence": 1, "constructed": 2, "regent": 1, "drying": 1, "lodge": 1, "acquired": 1, "suburban": 1, "brickfieldsan": 1, "providing": 1, "pond": 3, "materials": 1, "raft": 2, "equal": 1, "intimately": 1, "acquainted": 1, "superfluous": 1, "reluctant": 1, "accepting": 1, "proof": 1, "coolness": 1, "dodges": 1, "flattering": 1, "advances": 1, "youthful": 1, "legged": 1, "inevitable": 1, "interview": 1, "extremely": 3, "mostly": 1, "exclamatory": 1, "mono": 1, "syllabic": 1, "devoted": 2, "proficient": 1, "lea": 2, "clubs": 1, "afternoons": 1, "smart": 2, "handling": 1, "swamped": 1, "acquiring": 1, "prompt": 1, "admired": 1, "sixteen": 2, "kew": 3, "hiring": 1, "richmond": 1, "named": 1, "joskins": 4, "serpentine": 1, "tide": 1, "proceeded": 2, "select": 1, "oared": 1, "racing": 1, "outrigger": 3, "ardour": 1, "launched": 1, "cox": 3, "shoved": 1, "detail": 2, "disappear": 1, "magic": 1, "broadside": 1, "dipping": 1, "oar": 3, "entertainment": 1, "arch": 1, "renewed": 1, "sobs": 1, "prefers": 1, "eastbourne": 1, "flourishing": 1, "parade": 1, "nobility": 1, "gentry": 1, "secured": 1, "hired": 9, "fretful": 1, "vehement": 1, "striving": 1, "bane": 1, "straining": 1, "overtakes": 1, "annoy": 1, "overtake": 1, "himall": 1, "sublime": 1, "equanimity": 1, "ordeal": 1, "lesson": 2, "uppishness": 1, "youngster": 1, "twentieth": 1, "disentangles": 1, "explains": 1, "adapt": 1, "limited": 1, "capacity": 1, "devote": 1, "setting": 1, "moderate": 1, "flash": 2, "inspiration": 1, "answers": 1, "willingly": 1, "assisting": 1, "exchange": 1, "notnot": 1, "recovery": 1, "mutual": 1, "abuse": 1, "friendly": 1, "sympathetic": 1, "cheeky": 1, "plant": 1, "punter": 1, "unfortunately": 1, "clinging": 3, "undignified": 1, "lagging": 1, "precaution": 1, "scramble": 1, "streampossibly": 1, "lent": 1, "engaged": 2, "attracted": 2, "jacket": 2, "novice": 1, "spun": 1, "bets": 1, "outcome": 1, "exhibition": 1, "bounds": 1, "chaff": 1, "unmercifully": 1, "reprove": 1, "ridiculing": 1, "ribaldry": 1, "deriding": 1, "jeering": 1, "peppered": 1, "stale": 1, "jokes": 2, "private": 1, "belonging": 2, "perfectly": 1, "unintelligible": 1, "jibes": 1, "decency": 1, "deem": 1, "excused": 1, "boulogne": 1, "forcibly": 1, "whoever": 1, "hercules": 1, "unavailing": 1, "captor": 1, "released": 2, "regained": 1, "heartily": 3, "emerged": 1, "stammered": 1, "confusedly": 1, "relation": 1, "outright": 1, "toothough": 1, "rounders": 1, "sport": 1, "yare": 1, "reef": 2, "luff": 1, "luffed": 1, "hurricane": 1, "hectori": 1, "namewent": 1, "upside": 2, "funerals": 1, "boom": 1, "refused": 2, "hector": 2, "ships": 1, "wetted": 2, "sopping": 1, "vexing": 1, "downmore": 1, "sideways": 1, "likeand": 1, "painter": 1, "arriving": 1, "phenomenon": 1, "obstinacy": 1, "suicide": 1, "disappoint": 1, "exhausting": 1, "seafaring": 1, "lashed": 1, "main": 1, "jib": 1, "squalls": 1, "easiest": 1, "ended": 2, "contrived": 1, "embrace": 1, "travelled": 1, "sailed": 1, "heeled": 1, "righted": 1, "miracle": 1, "ploughed": 1, "according": 1, "bladder": 1, "surfeit": 1, "saila": 1, "sailand": 1, "fisherman": 5, "rescued": 1, "ignominious": 1, "tipping": 1, "considerable": 2, "xvi": 1, "shirk": 1, "streatley": 13, "linger": 2, "ethelred": 3, "anchored": 1, "warships": 1, "kennet": 1, "ravage": 1, "wessex": 2, "alfred": 2, "praying": 1, "parliament": 2, "plague": 2, "westminster": 1, "1625": 2, "courts": 1, "lawyers": 1, "parliamentary": 2, "besieged": 1, "essex": 1, "prince": 1, "james": 1, "benedictine": 1, "gaunt": 1, "blanche": 1, "continually": 2, "confoundedly": 1, "impertinent": 1, "tilehurst": 1, "mapledurham": 2, "hardwick": 1, "bowls": 1, "pangbourne": 6, "habitues": 1, "exhibitions": 1, "loose": 1, "unreasonable": 1, "surely": 2, "neared": 1, "blanched": 1, "prematurely": 1, "aged": 2, "stamp": 1, "pinch": 1, "poverty": 1, "uswe": 1, "coroners": 1, "courtssome": 1, "deceivedor": 1, "deceived": 2, "sinnedsome": 1, "thenand": 1, "closed": 1, "millstone": 1, "drudgery": 1, "procured": 1, "remainder": 1, "unitedly": 1, "bond": 1, "monotony": 2, "plainer": 1, "spectre": 1, "respectability": 1, "erring": 1, "outcast": 1, "unheeded": 2, "childhad": 1, "betraying": 1, "chocolate": 1, "bitterest": 1, "centred": 1, "hug": 1, "stabs": 1, "gall": 1, "shadowed": 1, "deeps": 1, "dusky": 1, "robe": 1, "sinned": 1, "thingssinned": 1, "sinners": 1, "woo": 1, "smiling": 3, "formed": 2, "barrier": 1, "contradict": 2, "affirm": 1, "statement": 2, "villages": 1, "choice": 1, "xvii": 1, "angling": 2, "conscientious": 2, "fisher": 1, "fishy": 2, "superintendence": 1, "failure": 2, "wearable": 1, "themwell": 1, "cleaner": 1, "dirt": 1, "washerwoman": 1, "excavating": 1, "abounds": 1, "pike": 2, "roach": 1, "dace": 2, "gudgeon": 2, "eels": 1, "minnows": 1, "district": 1, "angler": 6, "perch": 3, "shoals": 1, "itnot": 1, "thrower": 1, "gumption": 1, "imagination": 1, "shocker": 1, "reporter": 1, "invention": 1, "possess": 1, "ability": 1, "fabrication": 1, "tyro": 1, "circumstantial": 1, "embellishing": 1, "probability": 1, "scrupulousalmost": 1, "pedanticveracity": 1, "experienced": 1, "weighing": 1, "measuring": 1, "appropriates": 1, "puff": 1, "lets": 1, "brag": 1, "momentary": 1, "lull": 1, "removes": 1, "knocks": 1, "calmly": 1, "tinge": 1, "bitterness": 1, "refills": 1, "pause": 2, "continues": 1, "shouldn": 1, "literally": 1, "nothingexcept": 1, "rod": 3, "hourhalf": 1, "snap": 1, "sturgeon": 2, "surprisedi": 1, "astonishment": 1, "buggles": 1, "missus": 1, "listens": 1, "hauls": 1, "per": 3, "cent": 3, "add": 1, "threeat": 1, "increased": 1, "percentage": 1, "simplify": 1, "doubled": 2, "moderation": 1, "disadvantage": 1, "anglers": 1, "jealous": 1, "fishyou": 1, "foundation": 1, "lately": 1, "committee": 1, "association": 1, "adoption": 1, "older": 1, "opposed": 2, "sipping": 1, "indigestion": 1, "shave": 1, "pipeclaying": 1, "sincegeorge": 1, "parlour": 3, "trout": 16, "ensued": 1, "chimney": 1, "fascinated": 1, "monstrous": 1, "cod": 1, "uncommon": 1, "weighed": 3, "ounces": 1, "wur": 2, "minnow": 1, "remarkably": 1, "carrier": 2, "genial": 1, "lockleastways": 1, "thenone": 1, "whopper": 1, "aback": 1, "bleak": 1, "individual": 1, "comer": 1, "forgive": 1, "weperfect": 1, "neighbourhoodare": 1, "thingmost": 1, "guessing": 1, "scale": 1, "histories": 1, "immensely": 1, "bates": 1, "muggles": 1, "jones": 1, "billy": 1, "maunders": 1, "honest": 1, "plays": 1, "wag": 1, "bringing": 1, "whacking": 1, "astonishing": 1, "marvelled": 1, "alarm": 1, "fragmentsi": 1, "pieces": 3, "paris": 1, "xviii": 1, "photographed": 1, "dorchester": 6, "drowning": 1, "demoralizing": 1, "culham": 2, "extraordinarily": 1, "cleve": 1, "longest": 1, "teddington": 1, "eights": 1, "regretted": 1, "seeker": 1, "depths": 1, "sinking": 1, "strip": 1, "widens": 1, "prison": 1, "welcoming": 1, "eyed": 1, "287": 1, "exchanged": 1, "fairyland": 1, "speculative": 1, "photographer": 2, "hurriedly": 1, "ruffle": 1, "rakish": 1, "assuming": 1, "affability": 1, "fan": 1, "frowning": 1, "ours": 3, "agility": 1, "forehead": 1, "wistfulness": 1, "cynicism": 1, "eventful": 1, "rightat": 1, "altered": 1, "squinted": 1, "noses": 1, "pushed": 1, "stentorian": 1, "photo": 3, "woodwork": 1, "tilting": 1, "photograph": 3, "ordained": 1, "madly": 1, "undoubtedly": 1, "foreground": 1, "bits": 1, "surrounding": 2, "insignificant": 1, "paltry": 1, "compared": 1, "subscribe": 1, "bespoke": 1, "copies": 2, "rescinded": 1, "negative": 1, "unpleasantness": 1, "tenths": 1, "britons": 1, "evicted": 1, "replaced": 1, "fortifications": 2, "trace": 1, "sweeping": 1, "masons": 1, "halted": 1, "crumbled": 1, "romans": 2, "saxons": 1, "normans": 1, "walled": 1, "fortified": 1, "siege": 1, "fairfax": 1, "razed": 1, "hilly": 1, "varied": 1, "paddling": 1, "delightfully": 1, "drowsiness": 1, "caer": 1, "doren": 1, "recent": 1, "capital": 1, "nods": 1, "clifton": 4, "hampden": 1, "wonderfully": 1, "barley": 2, "mow": 2, "exception": 1, "quaintest": 1, "gables": 1, "timeyfied": 1, "modern": 2, "divinely": 1, "bump": 1, "unexpected": 1, "operation": 1, "impossibility": 1, "surprising": 1, "featherbed": 1, "monotonous": 1, "culhalm": 1, "lockthe": 1, "coldest": 1, "deepest": 1, "riverthe": 1, "improves": 1, "typical": 1, "smaller": 1, "orderquiet": 1, "eminently": 1, "desperately": 1, "prides": 1, "compare": 1, "doubtful": 1, "sanctified": 1, "brew": 1, "nowadays": 1, "nicholas": 1, "monument": 1, "blackwall": 1, "jane": 1, "helen": 1, "w": 3, "lee": 2, "1637": 1, "issue": 1, "loins": 1, "lacking": 1, "numbered": 1, "ninety": 1, "leefive": 1, "mayor": 1, "abingdonwas": 1, "benefactor": 1, "overcrowded": 1, "nuneham": 2, "courteney": 1, "visit": 1, "viewed": 1, "tuesdays": 1, "thursdays": 1, "pictures": 3, "curiosities": 1, "pool": 1, "lasher": 1, "undercurrent": 1, "obelisk": 2, "marks": 1, "diving": 1, "iffley": 4, "mill": 2, "favourite": 1, "brethren": 1, "disappointing": 1, "fairish": 1, "drives": 1, "college": 1, "exceptionally": 1, "irritable": 1, "mishaps": 1, "occur": 1, "indulgently": 1, "behave": 1, "mildest": 1, "gentlest": 1, "imaginable": 1, "unfortunate": 1, "sculler": 1, "brutally": 1, "amiable": 1, "demoralising": 1, "calmer": 1, "regret": 1, "xix": 1, "beauties": 1, "changes": 1, "aspects": 2, "yearnings": 1, "mournful": 2, "flight": 1, "constitutionally": 2, "whichever": 1, "squaring": 1, "contemplate": 1, "boatunless": 1, "hire": 2, "handled": 1, "rarely": 1, "sink": 1, "arrangementsor": 1, "allto": 1, "ornamental": 1, "airs": 1, "chiefone": 1, "recommendation": 1, "modest": 1, "likes": 1, "hides": 1, "boata": 1, "names": 1, "struggling": 1, "antediluvian": 1, "recently": 1, "carelessly": 1, "relic": 3, "coffin": 3, "relics": 1, "surmise": 1, "geologist": 1, "pooh": 1, "poohed": 1, "meanest": 1, "category": 1, "include": 1, "fossil": 2, "whale": 2, "evidences": 1, "proving": 1, "preglacial": 1, "pre": 1, "adamite": 1, "humorous": 1, "reward": 1, "persisted": 1, "sharply": 1, "tub": 1, "builder": 1, "boatwas": 1, "selected": 1, "whitewashed": 1, "tarredhad": 1, "distinguish": 1, "offended": 1, "stock": 1, "pasted": 1, "shabbier": 1, "prayers": 1, "loan": 1, "remnant": 1, "homeward": 1, "drizzle": 1, "riverwith": 1, "wavelets": 1, "gilding": 1, "beech": 1, "chasing": 1, "flinging": 1, "diamonds": 1, "wheels": 1, "kisses": 1, "wantoning": 1, "silvering": 1, "bridges": 1, "townlet": 1, "inlet": 1, "gleaming": 1, "gloryis": 1, "riverchill": 1, "ceaseless": 1, "weeping": 1, "shrouded": 1, "mists": 1, "vapour": 1, "reproachful": 1, "actions": 1, "neglectedis": 1, "regrets": 1, "melancholy": 1, "sunshine": 2, "enthusiastic": 1, "gipsy": 3, "storm": 1, "soberly": 1, "hoisted": 1, "paddle": 2, "poured": 1, "persistency": 1, "clammy": 1, "veal": 1, "apt": 1, "whitebait": 1, "cutlet": 1, "babbled": 1, "soles": 1, "sauce": 1, "nap": 1, "fourpencegeorge": 1, "cardsand": 1, "gambling": 1, "breeds": 1, "revenge": 1, "saddest": 1, "volunteers": 1, "aldershot": 1, "cripple": 1, "introduce": 1, "sciatica": 1, "fevers": 1, "chills": 1, "lung": 1, "frolicksome": 1, "vein": 1, "extracted": 1, "weep": 1, "yearnful": 1, "unutterable": 1, "jaw": 1, "abandon": 1, "rendering": 1, "bedthat": 1, "undressed": 1, "slumber": 1, "pour": 1, "mackintoshes": 1, "usi": 1, "myselfmade": 1, "attempts": 1, "expressing": 1, "sentiments": 1, "unnecessary": 1, "enjoyment": 2, "climate": 1, "disastrous": 1, "prospect": 1, "finish": 1, "almanac": 1, "skirt": 1, "alhambra": 2, "venturing": 1, "survey": 1, "311": 1, "unconsciously": 1, "contract": 1, "deaths": 1, "casting": 1, "malevolence": 1, "mention": 1, "figures": 1, "shamed": 1, "stealthily": 1, "gaudy": 1, "mackintosh": 1, "instructions": 1, "unforeseen": 1, "paddington": 1, "restaurant": 2, "leicester": 1, "presenting": 1, "paybox": 1, "informed": 1, "renowned": 1, "contortionists": 1, "himalaya": 1, "greater": 1, "bronzed": 1, "countenances": 1, "cynosure": 1, "awaiting": 1, "burgundy": 1, "sauces": 1, "napkins": 2, "loaves": 1, "welcome": 1, "pegged": 1, "quaffed": 1, "carelesslywhen": 1, "critically": 1, "smoky": 1, "hitherto": 1, "dowhen": 1, "curtain": 1, "glistened": 1, "darkly": 1, "lamps": 1, "flickered": 1, "gust": 1, "puddles": 1, "trickled": 1, "spouts": 1, "gutters": 1, "wayfarers": 1, "dripping": 1, "skirts": 1, "thamesbut": 1, "hind": 1, "concurrence": 1}'

The above is the number of times each word appears in the corpus. As you can see, the word counts dictionaries in the config are serialized into plain JSON. The loads() method in the Python library json can be used to convert this JSON string into a dictionary.

In [12]:

# Save the word_counts as a python dictionary
import json
word_counts = json.loads(tokenizer_config['word_counts'])

The word index is derived from the word_counts.

In [13]:

# View the word_index entry
tokenizer_config['word_index']

Out[13]:

'{"<UNK>": 1, "the": 2, "and": 3, "to": 4, "a": 5, "of": 6, "it": 7, "i": 8, "in": 9, "that": 10, "he": 11, "we": 12, "was": 13, "you": 14, "had": 15, "for": 16, "at": 17, "on": 18, "with": 19, "up": 20, "they": 21, "is": 22, "as": 23, "not": 24, "his": 25, "said": 26, "but": 27, "would": 28, "all": 29, "s": 30, "have": 31, "him": 32, "there": 33, "be": 34, "harris": 35, "george": 36, "out": 37, "t": 38, "so": 39, "then": 40, "when": 41, "them": 42, "one": 43, "were": 44, "about": 45, "us": 46, "from": 47, "by": 48, "what": 49, "our": 50, "boat": 51, "down": 52, "my": 53, "been": 54, "if": 55, "an": 56, "me": 57, "this": 58, "or": 59, "are": 60, "get": 61, "river": 62, "man": 63, "got": 64, "into": 65, "did": 66, "could": 67, "who": 68, "do": 69, "over": 70, "round": 71, "time": 72, "more": 73, "go": 74, "other": 75, "old": 76, "very": 77, "like": 78, "no": 79, "two": 80, "thought": 81, "their": 82, "back": 83, "know": 84, "never": 85, "little": 86, "can": 87, "come": 88, "after": 89, "some": 90, "she": 91, "good": 92, "way": 93, "which": 94, "went": 95, "now": 96, "her": 97, "upon": 98, "seemed": 99, "see": 100, "your": 101, "don": 102, "has": 103, "day": 104, "any": 105, "came": 106, "should": 107, "how": 108, "only": 109, "thing": 110, "its": 111, "before": 112, "people": 113, "water": 114, "put": 115, "take": 116, "will": 117, "off": 118, "away": 119, "where": 120, "picture": 121, "just": 122, "well": 123, "first": 124, "half": 125, "three": 126, "than": 127, "going": 128, "night": 129, "oh": 130, "much": 131, "quite": 132, "while": 133, "want": 134, "say": 135, "last": 136, "why": 137, "made": 138, "work": 139, "too": 140, "again": 141, "morning": 142, "things": 143, "ever": 144, "took": 145, "being": 146, "another": 147, "right": 148, "think": 149, "great": 150, "lock": 151, "montmorency": 152, "each": 153, "himself": 154, "once": 155, "young": 156, "myself": 157, "such": 158, "must": 159, "boy": 160, "men": 161, "side": 162, "those": 163, "told": 164, "left": 165, "make": 166, "bank": 167, "through": 168, "bit": 169, "look": 170, "without": 171, "looked": 172, "here": 173, "head": 174, "long": 175, "life": 176, "every": 177, "always": 178, "house": 179, "sat": 180, "bed": 181, "looking": 182, "five": 183, "against": 184, "under": 185, "might": 186, "does": 187, "place": 188, "let": 189, "same": 190, "asked": 191, "room": 192, "tell": 193, "didn": 194, "really": 195, "hand": 196, "most": 197, "end": 198, "knew": 199, "feel": 200, "give": 201, "enough": 202, "course": 203, "still": 204, "wanted": 205, "done": 206, "whole": 207, "minutes": 208, "four": 209, "anything": 210, "saw": 211, "felt": 212, "sort": 213, "hour": 214, "both": 215, "rather": 216, "however": 217, "matter": 218, "ten": 219, "something": 220, "supper": 221, "many": 222, "found": 223, "caught": 224, "dog": 225, "these": 226, "because": 227, "idea": 228, "evening": 229, "until": 230, "even": 231, "moment": 232, "tried": 233, "m": 234, "light": 235, "fish": 236, "began": 237, "years": 238, "days": 239, "may": 240, "world": 241, "town": 242, "friend": 243, "replied": 244, "line": 245, "doing": 246, "am": 247, "turned": 248, "find": 249, "nothing": 250, "gone": 251, "says": 252, "stop": 253, "song": 254, "home": 255, "ve": 256, "along": 257, "stream": 258, "mind": 259, "past": 260, "gave": 261, "ll": 262, "own": 263, "air": 264, "cold": 265, "full": 266, "heard": 267, "getting": 268, "tea": 269, "sit": 270, "sleep": 271, "taken": 272, "having": 273, "early": 274, "sight": 275, "behind": 276, "keep": 277, "try": 278, "miles": 279, "next": 280, "between": 281, "part": 282, "far": 283, "everything": 284, "used": 285, "better": 286, "across": 287, "till": 288, "wind": 289, "few": 290, "middle": 291, "sitting": 292, "reading": 293, "six": 294, "happened": 295, "ourselves": 296, "pretty": 297, "tow": 298, "started": 299, "mile": 300, "set": 301, "seen": 302, "nearly": 303, "hear": 304, "pass": 305, "trip": 306, "face": 307, "small": 308, "d": 309, "dogs": 310, "near": 311, "afterwards": 312, "door": 313, "sea": 314, "stand": 315, "met": 316, "themselves": 317, "ought": 318, "fine": 319, "above": 320, "friends": 321, "lady": 322, "times": 323, "walk": 324, "strange": 325, "couldn": 326, "sir": 327, "given": 328, "nose": 329, "eyes": 330, "rain": 331, "also": 332, "trying": 333, "together": 334, "dark": 335, "else": 336, "pulled": 337, "watch": 338, "kept": 339, "spot": 340, "sweet": 341, "pleasant": 342, "hard": 343, "care": 344, "wet": 345, "coming": 346, "thames": 347, "seem": 348, "itself": 349, "soon": 350, "call": 351, "stood": 352, "breakfast": 353, "nobody": 354, "yes": 355, "though": 356, "new": 357, "pull": 358, "re": 359, "nature": 360, "launch": 361, "chapter": 362, "hundred": 363, "week": 364, "seems": 365, "case": 366, "able": 367, "sure": 368, "waiting": 369, "land": 370, "whether": 371, "during": 372, "sad": 373, "makes": 374, "wrong": 375, "somebody": 376, "cat": 377, "high": 378, "clothes": 379, "top": 380, "passed": 381, "steam": 382, "reached": 383, "lines": 384, "deep": 385, "point": 386, "remember": 387, "fancy": 388, "evidently": 389, "shall": 390, "hours": 391, "further": 392, "mrs": 393, "couple": 394, "fellow": 395, "held": 396, "expression": 397, "feet": 398, "ago": 399, "pie": 400, "help": 401, "trouble": 402, "number": 403, "bridge": 404, "boats": 405, "sail": 406, "bottom": 407, "twenty": 408, "start": 409, "weather": 410, "ill": 411, "turn": 412, "hold": 413, "ready": 414, "brought": 415, "drew": 416, "begin": 417, "kingston": 418, "wake": 419, "either": 420, "sing": 421, "voice": 422, "king": 423, "silence": 424, "mother": 425, "street": 426, "called": 427, "upset": 428, "tom": 429, "chair": 430, "word": 431, "red": 432, "woman": 433, "marlow": 434, "hamper": 435, "girls": 436, "best": 437, "rowing": 438, "comic": 439, "view": 440, "heart": 441, "stopped": 442, "straight": 443, "fact": 444, "family": 445, "understand": 446, "followed": 447, "general": 448, "kind": 449, "often": 450, "simple": 451, "o": 452, "worth": 453, "board": 454, "around": 455, "comes": 456, "spirit": 457, "suddenly": 458, "legs": 459, "poor": 460, "wall": 461, "kettle": 462, "german": 463, "party": 464, "bow": 465, "father": 466, "trout": 467, "seven": 468, "bad": 469, "since": 470, "cannot": 471, "living": 472, "change": 473, "suggested": 474, "ask": 475, "bag": 476, "afternoon": 477, "goes": 478, "quiet": 479, "arms": 480, "fair": 481, "fixed": 482, "ah": 483, "glass": 484, "row": 485, "yards": 486, "altogether": 487, "show": 488, "drink": 489, "boating": 490, "church": 491, "cheeses": 492, "taking": 493, "real": 494, "opposite": 495, "village": 496, "towed": 497, "rest": 498, "mean": 499, "nervous": 500, "bright": 501, "live": 502, "known": 503, "mine": 504, "finished": 505, "open": 506, "hotel": 507, "among": 508, "sometimes": 509, "reach": 510, "close": 511, "towards": 512, "afraid": 513, "answered": 514, "run": 515, "low": 516, "singing": 517, "drop": 518, "believe": 519, "shout": 520, "business": 521, "inn": 522, "lost": 523, "language": 524, "canvas": 525, "oil": 526, "gentleman": 527, "later": 528, "inside": 529, "become": 530, "lovely": 531, "sculls": 532, "laughing": 533, "beautiful": 534, "making": 535, "stroke": 536, "fishing": 537, "island": 538, "john": 539, "children": 540, "exactly": 541, "forget": 542, "determined": 543, "somehow": 544, "less": 545, "crowd": 546, "eight": 547, "dead": 548, "deal": 549, "within": 550, "yet": 551, "position": 552, "leave": 553, "seeing": 554, "summer": 555, "bread": 556, "surprised": 557, "fellows": 558, "meet": 559, "slowly": 560, "grass": 561, "big": 562, "laugh": 563, "white": 564, "corner": 565, "heavy": 566, "mad": 567, "steadily": 568, "lay": 569, "wood": 570, "thirty": 571, "hang": 572, "hammer": 573, "rule": 574, "anybody": 575, "use": 576, "human": 577, "notice": 578, "dress": 579, "easy": 580, "oak": 581, "broken": 582, "added": 583, "strong": 584, "noticed": 585, "oxford": 586, "wallingford": 587, "abbey": 588, "streatley": 589, "front": 590, "eye": 591, "happy": 592, "immediately": 593, "present": 594, "interest": 595, "lit": 596, "everybody": 597, "clock": 598, "answer": 599, "london": 600, "train": 601, "second": 602, "respectable": 603, "except": 604, "outside": 605, "mere": 606, "instead": 607, "follow": 608, "third": 609, "death": 610, "sent": 611, "bring": 612, "podger": 613, "different": 614, "blue": 615, "packed": 616, "died": 617, "thinking": 618, "towing": 619, "late": 620, "eggs": 621, "quarter": 622, "nine": 623, "passing": 624, "suppose": 625, "hat": 626, "lie": 627, "art": 628, "diddle": 629, "pole": 630, "least": 631, "punt": 632, "hardly": 633, "read": 634, "generally": 635, "hadn": 636, "forty": 637, "walking": 638, "practice": 639, "scene": 640, "love": 641, "boys": 642, "pipe": 643, "wish": 644, "beef": 645, "plenty": 646, "indeed": 647, "save": 648, "won": 649, "decided": 650, "following": 651, "wild": 652, "silent": 653, "cry": 654, "woods": 655, "black": 656, "common": 657, "below": 658, "proper": 659, "earth": 660, "gentlemen": 661, "fight": 662, "settled": 663, "uncle": 664, "clear": 665, "merely": 666, "music": 667, "pair": 668, "hope": 669, "boots": 670, "question": 671, "pan": 672, "country": 673, "woke": 674, "whose": 675, "soul": 676, "wait": 677, "biggs": 678, "neighbourhood": 679, "slept": 680, "catch": 681, "sculling": 682, "funny": 683, "pulling": 684, "tin": 685, "talking": 686, "stage": 687, "interesting": 688, "minute": 689, "stuck": 690, "telling": 691, "beer": 692, "state": 693, "weak": 694, "cheese": 695, "agreed": 696, "dream": 697, "quaint": 698, "sound": 699, "short": 700, "turning": 701, "lunch": 702, "job": 703, "nor": 704, "haven": 705, "wished": 706, "bath": 707, "body": 708, "almost": 709, "struck": 710, "waters": 711, "story": 712, "difficult": 713, "fire": 714, "sufficient": 715, "method": 716, "ground": 717, "dozen": 718, "paper": 719, "standing": 720, "expect": 721, "mast": 722, "houses": 723, "stern": 724, "wonder": 725, "awful": 726, "conversation": 727, "threw": 728, "waited": 729, "style": 730, "jolly": 731, "break": 732, "carved": 733, "hampton": 734, "picturesque": 735, "road": 736, "walls": 737, "stone": 738, "map": 739, "judge": 740, "hair": 741, "banjo": 742, "subject": 743, "pianist": 744, "goring": 745, "spoke": 746, "lying": 747, "mustard": 748, "months": 749, "st": 750, "expected": 751, "opinion": 752, "certain": 753, "looks": 754, "explained": 755, "whatever": 756, "none": 757, "somewhat": 758, "sunny": 759, "meant": 760, "yourself": 761, "rolled": 762, "smile": 763, "umbrella": 764, "step": 765, "dry": 766, "worked": 767, "landing": 768, "curious": 769, "easily": 770, "dangerous": 771, "gets": 772, "fresh": 773, "child": 774, "grows": 775, "thoughts": 776, "trees": 777, "knows": 778, "whisky": 779, "banks": 780, "stove": 781, "rich": 782, "jam": 783, "therefore": 784, "large": 785, "longer": 786, "thus": 787, "coat": 788, "doesn": 789, "nail": 790, "pride": 791, "empty": 792, "green": 793, "stretch": 794, "french": 795, "washing": 796, "frying": 797, "quietly": 798, "carrying": 799, "result": 800, "fell": 801, "noble": 802, "ain": 803, "exciting": 804, "watched": 805, "cried": 806, "roared": 807, "grand": 808, "rugs": 809, "whom": 810, "steering": 811, "c\\u00e6sar": 812, "faces": 813, "school": 814, "dull": 815, "length": 816, "chance": 817, "success": 818, "continued": 819, "boards": 820, "running": 821, "takes": 822, "push": 823, "herr": 824, "boschen": 825, "manner": 826, "weir": 827, "henry": 828, "hook": 829, "henley": 830, "stew": 831, "play": 832, "swans": 833, "jack": 834, "hired": 835, "liver": 836, "feeling": 837, "extraordinary": 838, "order": 839, "knowing": 840, "born": 841, "grew": 842, "account": 843, "feels": 844, "dear": 845, "mistake": 846, "table": 847, "pipes": 848, "cousin": 849, "century": 850, "pounds": 851, "weeks": 852, "reply": 853, "lived": 854, "managed": 855, "purpose": 856, "rough": 857, "butter": 858, "tone": 859, "city": 860, "sun": 861, "tent": 862, "playing": 863, "voices": 864, "die": 865, "rising": 866, "damp": 867, "simply": 868, "fool": 869, "isn": 870, "dressed": 871, "ladies": 872, "killed": 873, "shillings": 874, "bathing": 875, "mr": 876, "spring": 877, "blow": 878, "clean": 879, "iron": 880, "throw": 881, "name": 882, "eat": 883, "huge": 884, "liked": 885, "trousers": 886, "married": 887, "shop": 888, "rose": 889, "laughed": 890, "yours": 891, "please": 892, "rush": 893, "wife": 894, "speak": 895, "touch": 896, "others": 897, "window": 898, "hands": 899, "twelve": 900, "seat": 901, "appeared": 902, "tail": 903, "lot": 904, "hurry": 905, "vision": 906, "joined": 907, "thoughtful": 908, "slipped": 909, "english": 910, "maze": 911, "year": 912, "calm": 913, "peaceful": 914, "queen": 915, "famous": 916, "desire": 917, "herself": 918, "tree": 919, "shirt": 920, "spare": 921, "surprise": 922, "played": 923, "alone": 924, "rushed": 925, "scull": 926, "sorry": 927, "islands": 928, "barons": 929, "need": 930, "carried": 931, "various": 932, "fever": 933, "class": 934, "wondered": 935, "shut": 936, "stick": 937, "wasn": 938, "pocket": 939, "ran": 940, "bitter": 941, "beyond": 942, "stomach": 943, "charge": 944, "naturally": 945, "faint": 946, "money": 947, "shore": 948, "hearted": 949, "return": 950, "liverpool": 951, "exercise": 952, "railway": 953, "dinner": 954, "meat": 955, "boiled": 956, "hot": 957, "hi": 958, "thrown": 959, "lean": 960, "suit": 961, "credit": 962, "scenery": 963, "discussed": 964, "camping": 965, "shadows": 966, "palace": 967, "talk": 968, "words": 969, "fall": 970, "asleep": 971, "beneath": 972, "tears": 973, "regarded": 974, "places": 975, "fix": 976, "gives": 977, "ass": 978, "bill": 979, "evident": 980, "reason": 981, "happen": 982, "march": 983, "irish": 984, "cut": 985, "handkerchief": 986, "spent": 987, "craft": 988, "ha": 989, "crown": 990, "beg": 991, "brush": 992, "tooth": 993, "anywhere": 994, "pack": 995, "twice": 996, "mid": 997, "wash": 998, "dirty": 999, "rate": 1000, "station": 1001, "already": 1002, "gathered": 1003, "keeping": 1004, "cheerful": 1005, "spread": 1006, "headed": 1007, "wrapped": 1008, "edge": 1009, "stepped": 1010, "wouldn": 1011, "exclaimed": 1012, "particularly": 1013, "whenever": 1014, "beds": 1015, "luggage": 1016, "shouted": 1017, "certainly": 1018, "fighting": 1019, "silly": 1020, "picnic": 1021, "matters": 1022, "pointed": 1023, "blood": 1024, "lies": 1025, "windsor": 1026, "somewhere": 1027, "court": 1028, "glorious": 1029, "dainty": 1030, "sunlight": 1031, "roman": 1032, "perhaps": 1033, "rude": 1034, "firmly": 1035, "quickly": 1036, "drawing": 1037, "remarkable": 1038, "lovers": 1039, "gas": 1040, "path": 1041, "ghosts": 1042, "fear": 1043, "keeper": 1044, "tomb": 1045, "graves": 1046, "folk": 1047, "offer": 1048, "coats": 1049, "shock": 1050, "tombs": 1051, "park": 1052, "songs": 1053, "chorus": 1054, "slossenn": 1055, "backwater": 1056, "laid": 1057, "thinks": 1058, "glance": 1059, "distance": 1060, "hers": 1061, "staines": 1062, "effort": 1063, "pain": 1064, "hall": 1065, "washed": 1066, "worlds": 1067, "sailing": 1068, "broke": 1069, "landlord": 1070, "locks": 1071, "bisham": 1072, "bull": 1073, "potatoes": 1074, "boatman": 1075, "mud": 1076, "smoking": 1077, "medical": 1078, "suffering": 1079, "book": 1080, "leaves": 1081, "study": 1082, "fearful": 1083, "list": 1084, "fortnight": 1085, "walked": 1086, "wants": 1087, "commonplace": 1088, "hit": 1089, "chest": 1090, "stuff": 1091, "effect": 1092, "piece": 1093, "knocked": 1094, "duty": 1095, "produced": 1096, "forgotten": 1097, "wicked": 1098, "monday": 1099, "bosom": 1100, "enjoy": 1101, "saturday": 1102, "food": 1103, "brother": 1104, "meal": 1105, "pound": 1106, "tied": 1107, "leaning": 1108, "overboard": 1109, "loved": 1110, "centre": 1111, "inns": 1112, "golden": 1113, "dim": 1114, "grey": 1115, "stars": 1116, "musical": 1117, "fond": 1118, "glad": 1119, "nice": 1120, "becomes": 1121, "indignantly": 1122, "methylated": 1123, "sleeping": 1124, "underneath": 1125, "events": 1126, "fox": 1127, "learned": 1128, "arrangements": 1129, "square": 1130, "j": 1131, "girl": 1132, "leg": 1133, "string": 1134, "possible": 1135, "allow": 1136, "women": 1137, "danger": 1138, "pleasure": 1139, "plain": 1140, "god": 1141, "comfortable": 1142, "hoops": 1143, "paraffine": 1144, "ruined": 1145, "yard": 1146, "buried": 1147, "field": 1148, "bacon": 1149, "built": 1150, "presence": 1151, "brown": 1152, "platform": 1153, "falling": 1154, "solemn": 1155, "appearance": 1156, "bags": 1157, "instant": 1158, "returned": 1159, "wonderful": 1160, "person": 1161, "working": 1162, "boot": 1163, "eighteen": 1164, "cup": 1165, "perfect": 1166, "succeeded": 1167, "flew": 1168, "landed": 1169, "local": 1170, "instrument": 1171, "steady": 1172, "wretched": 1173, "carefully": 1174, "guide": 1175, "barges": 1176, "saxon": 1177, "minded": 1178, "battle": 1179, "gates": 1180, "windows": 1181, "speaking": 1182, "weird": 1183, "gold": 1184, "runs": 1185, "draw": 1186, "cast": 1187, "nearer": 1188, "beginning": 1189, "accident": 1190, "taste": 1191, "thomas": 1192, "skulls": 1193, "em": 1194, "relations": 1195, "sur": 1196, "barge": 1197, "grim": 1198, "viii": 1199, "required": 1200, "giving": 1201, "verse": 1202, "prelude": 1203, "jury": 1204, "sees": 1205, "truth": 1206, "murmured": 1207, "prepared": 1208, "abingdon": 1209, "flat": 1210, "double": 1211, "saved": 1212, "tired": 1213, "stretched": 1214, "gently": 1215, "maidenhead": 1216, "hitcher": 1217, "sadness": 1218, "emily": 1219, "drowned": 1220, "leading": 1221, "month": 1222, "knife": 1223, "drawn": 1224, "anne": 1225, "stands": 1226, "launches": 1227, "peace": 1228, "sonning": 1229, "poodle": 1230, "wargrave": 1231, "bagpipes": 1232, "tune": 1233, "punting": 1234, "pangbourne": 1235, "angler": 1236, "dorchester": 1237, "original": 1238, "conclusion": 1239, "disease": 1240, "severe": 1241, "housemaid": 1242, "knee": 1243, "feelings": 1244, "appear": 1245, "seized": 1246, "ordinary": 1247, "hampers": 1248, "1": 1249, "beefsteak": 1250, "directions": 1251, "instance": 1252, "anxious": 1253, "putting": 1254, "nineteenth": 1255, "beat": 1256, "captain": 1257, "cook": 1258, "law": 1259, "ship": 1260, "hungry": 1261, "cream": 1262, "enthusiasm": 1263, "oily": 1264, "quick": 1265, "sick": 1266, "discovered": 1267, "move": 1268, "fooling": 1269, "camp": 1270, "rushes": 1271, "filled": 1272, "chat": 1273, "tales": 1274, "sings": 1275, "thousand": 1276, "listen": 1277, "sister": 1278, "ere": 1279, "loving": 1280, "scotch": 1281, "brilliant": 1282, "practical": 1283, "beautifully": 1284, "direction": 1285, "heap": 1286, "ruins": 1287, "article": 1288, "bottle": 1289, "struggle": 1290, "kill": 1291, "jim": 1292, "terrier": 1293, "snatched": 1294, "neck": 1295, "maybe": 1296, "warm": 1297, "write": 1298, "aunt": 1299, "send": 1300, "size": 1301, "candle": 1302, "piano": 1303, "hoped": 1304, "mess": 1305, "labour": 1306, "comfort": 1307, "useless": 1308, "lumber": 1309, "twopence": 1310, "anxiety": 1311, "someone": 1312, "pardon": 1313, "forgot": 1314, "cover": 1315, "bathe": 1316, "contrary": 1317, "enjoyed": 1318, "ate": 1319, "finally": 1320, "experience": 1321, "miserable": 1322, "shoes": 1323, "companion": 1324, "horse": 1325, "cab": 1326, "merry": 1327, "leaving": 1328, "carriage": 1329, "moments": 1330, "baby": 1331, "steps": 1332, "bury": 1333, "journey": 1334, "possibly": 1335, "regard": 1336, "rid": 1337, "mixed": 1338, "lemonade": 1339, "gladstone": 1340, "floor": 1341, "packing": 1342, "entirely": 1343, "aside": 1344, "irritating": 1345, "chuckle": 1346, "upstairs": 1347, "breaking": 1348, "object": 1349, "natural": 1350, "preferred": 1351, "throwing": 1352, "downstairs": 1353, "bar": 1354, "attention": 1355, "became": 1356, "lower": 1357, "gloomy": 1358, "ye": 1359, "cheerily": 1360, "finding": 1361, "promptly": 1362, "grocer": 1363, "moving": 1364, "posts": 1365, "luck": 1366, "eleven": 1367, "porter": 1368, "anyhow": 1369, "shot": 1370, "history": 1371, "wooded": 1372, "distant": 1373, "england": 1374, "moonlight": 1375, "gay": 1376, "stairs": 1377, "lives": 1378, "forth": 1379, "covered": 1380, "blame": 1381, "doubt": 1382, "badly": 1383, "spite": 1384, "beauty": 1385, "age": 1386, "china": 1387, "ancient": 1388, "especially": 1389, "consequence": 1390, "creeping": 1391, "fifty": 1392, "hill": 1393, "picked": 1394, "impossible": 1395, "climbed": 1396, "wandered": 1397, "fashion": 1398, "costume": 1399, "opportunity": 1400, "vexed": 1401, "worn": 1402, "lively": 1403, "thick": 1404, "fun": 1405, "leant": 1406, "grave": 1407, "burst": 1408, "everyone": 1409, "cool": 1410, "steer": 1411, "stay": 1412, "trial": 1413, "lord": 1414, "kindly": 1415, "laughter": 1416, "flesh": 1417, "tittered": 1418, "violent": 1419, "bend": 1420, "fought": 1421, "cap": 1422, "excitement": 1423, "season": 1424, "skiff": 1425, "unaccountable": 1426, "ends": 1427, "fault": 1428, "dropped": 1429, "curses": 1430, "pace": 1431, "besides": 1432, "plan": 1433, "stopping": 1434, "twilight": 1435, "soft": 1436, "strength": 1437, "pillow": 1438, "joe": 1439, "whistle": 1440, "knights": 1441, "joy": 1442, "scrambled": 1443, "wondering": 1444, "passes": 1445, "bear": 1446, "uninteresting": 1447, "delightful": 1448, "unable": 1449, "cats": 1450, "basket": 1451, "avoid": 1452, "fisherman": 1453, "objection": 1454, "william": 1455, "fits": 1456, "particular": 1457, "diseases": 1458, "fairly": 1459, "interested": 1460, "learnt": 1461, "concerned": 1462, "hurt": 1463, "grasping": 1464, "pulse": 1465, "gain": 1466, "brief": 1467, "remains": 1468, "opened": 1469, "chemist": 1470, "handed": 1471, "stores": 1472, "sharp": 1473, "rug": 1474, "clever": 1475, "poppets": 1476, "supposed": 1477, "tart": 1478, "complete": 1479, "throughout": 1480, "usually": 1481, "nook": 1482, "noisy": 1483, "objected": 1484, "wave": 1485, "deck": 1486, "friday": 1487, "sunday": 1488, "ticket": 1489, "eventually": 1490, "pay": 1491, "beforehand": 1492, "hearty": 1493, "contented": 1494, "strawberries": 1495, "neither": 1496, "odour": 1497, "mingled": 1498, "ladder": 1499, "feeble": 1500, "thin": 1501, "biscuits": 1502, "gazed": 1503, "mystery": 1504, "thousands": 1505, "coffee": 1506, "headache": 1507, "forward": 1508, "minds": 1509, "including": 1510, "appetite": 1511, "lodging": 1512, "cheap": 1513, "sensible": 1514, "suggestion": 1515, "bally": 1516, "plans": 1517, "pleasures": 1518, "nights": 1519, "meeting": 1520, "arranged": 1521, "free": 1522, "clouds": 1523, "dying": 1524, "rear": 1525, "waving": 1526, "wings": 1527, "stillness": 1528, "sung": 1529, "silver": 1530, "knock": 1531, "ashamed": 1532, "led": 1533, "arm": 1534, "likely": 1535, "inches": 1536, "pouring": 1537, "stupid": 1538, "roar": 1539, "savage": 1540, "haul": 1541, "explain": 1542, "views": 1543, "swearing": 1544, "impression": 1545, "smothered": 1546, "begins": 1547, "replies": 1548, "inclined": 1549, "revel": 1550, "imagine": 1551, "angel": 1552, "dragged": 1553, "unknown": 1554, "rats": 1555, "disreputable": 1556, "hotels": 1557, "thirsty": 1558, "hats": 1559, "backs": 1560, "reminds": 1561, "level": 1562, "maria": 1563, "heads": 1564, "slip": 1565, "yell": 1566, "fuss": 1567, "hole": 1568, "midnight": 1569, "reaches": 1570, "sufficiently": 1571, "yellow": 1572, "lamp": 1573, "soap": 1574, "drawers": 1575, "towel": 1576, "stones": 1577, "sprawling": 1578, "talked": 1579, "travelling": 1580, "pot": 1581, "oozed": 1582, "rudder": 1583, "blew": 1584, "desert": 1585, "sunset": 1586, "lonely": 1587, "apple": 1588, "owner": 1589, "stout": 1590, "crowded": 1591, "yelled": 1592, "miss": 1593, "bought": 1594, "attached": 1595, "queried": 1596, "smell": 1597, "holiday": 1598, "roof": 1599, "husband": 1600, "slap": 1601, "pies": 1602, "piled": 1603, "shook": 1604, "10": 1605, "plates": 1606, "excited": 1607, "mysterious": 1608, "flying": 1609, "wasted": 1610, "encouragement": 1611, "sin": 1612, "prefer": 1613, "forecast": 1614, "waterloo": 1615, "south": 1616, "starting": 1617, "mouth": 1618, "sunk": 1619, "ear": 1620, "anyone": 1621, "invited": 1622, "ghastly": 1623, "yesterday": 1624, "paying": 1625, "fancied": 1626, "harder": 1627, "notion": 1628, "apparently": 1629, "period": 1630, "cross": 1631, "catching": 1632, "eased": 1633, "asking": 1634, "remarks": 1635, "streets": 1636, "orange": 1637, "blazer": 1638, "fit": 1639, "public": 1640, "stuffed": 1641, "watching": 1642, "build": 1643, "grown": 1644, "proprietor": 1645, "ceiling": 1646, "relief": 1647, "average": 1648, "bother": 1649, "sandford": 1650, "lad": 1651, "rows": 1652, "grow": 1653, "teeth": 1654, "stiff": 1655, "dug": 1656, "odd": 1657, "colour": 1658, "cracked": 1659, "temper": 1660, "shades": 1661, "frightened": 1662, "theory": 1663, "sang": 1664, "confused": 1665, "huddled": 1666, "rushing": 1667, "plate": 1668, "dresses": 1669, "affords": 1670, "pity": 1671, "seats": 1672, "arrangement": 1673, "churchyard": 1674, "wounds": 1675, "restful": 1676, "winding": 1677, "blessed": 1678, "tender": 1679, "nonsense": 1680, "tones": 1681, "conduct": 1682, "lunched": 1683, "commenced": 1684, "intensely": 1685, "evil": 1686, "worse": 1687, "service": 1688, "note": 1689, "jerk": 1690, "blest": 1691, "warning": 1692, "pinafore": 1693, "join": 1694, "tries": 1695, "sense": 1696, "emperor": 1697, "serious": 1698, "humour": 1699, "shady": 1700, "walton": 1701, "fields": 1702, "winds": 1703, "showing": 1704, "sensation": 1705, "seconds": 1706, "careless": 1707, "connection": 1708, "properly": 1709, "wildly": 1710, "drifted": 1711, "indignant": 1712, "frantic": 1713, "floating": 1714, "shouts": 1715, "cookham": 1716, "fate": 1717, "recover": 1718, "precise": 1719, "noise": 1720, "weary": 1721, "jerked": 1722, "bless": 1723, "comfortably": 1724, "fastened": 1725, "sockets": 1726, "crept": 1727, "knight": 1728, "castle": 1729, "unto": 1730, "darkness": 1731, "earthly": 1732, "grace": 1733, "received": 1734, "g": 1735, "policeman": 1736, "suspicion": 1737, "dismal": 1738, "police": 1739, "true": 1740, "howl": 1741, "pluck": 1742, "ale": 1743, "breeze": 1744, "shake": 1745, "liberty": 1746, "current": 1747, "kent": 1748, "edward": 1749, "earl": 1750, "datchet": 1751, "honeysuckle": 1752, "wore": 1753, "stag": 1754, "manor": 1755, "fly": 1756, "clay": 1757, "fishermen": 1758, "medmenham": 1759, "monks": 1760, "danes": 1761, "heaven": 1762, "lobby": 1763, "gentle": 1764, "satisfactory": 1765, "peas": 1766, "gaze": 1767, "skill": 1768, "shiplake": 1769, "potato": 1770, "efforts": 1771, "strangers": 1772, "swan": 1773, "count": 1774, "appears": 1775, "joskins": 1776, "clifton": 1777, "iffley": 1778, "victim": 1779, "motion": 1780, "majority": 1781, "symptoms": 1782, "therein": 1783, "form": 1784, "british": 1785, "slight": 1786, "borne": 1787, "typhoid": 1788, "commence": 1789, "concluded": 1790, "students": 1791, "examine": 1792, "beating": 1793, "tongue": 1794, "wreck": 1795, "doctor": 1796, "wrote": 1797, "folded": 1798, "combined": 1799, "11": 1800, "chief": 1801, "advanced": 1802, "box": 1803, "powerful": 1804, "smiled": 1805, "swallow": 1806, "onions": 1807, "unusual": 1808, "glasses": 1809, "discussion": 1810, "health": 1811, "actually": 1812, "depression": 1813, "system": 1814, "absence": 1815, "necessity": 1816, "described": 1817, "sheet": 1818, "strongly": 1819, "tuesday": 1820, "gunwale": 1821, "offered": 1822, "tremendous": 1823, "sold": 1824, "western": 1825, "coast": 1826, "steward": 1827, "arrange": 1828, "eating": 1829, "ropes": 1830, "greeted": 1831, "blameless": 1832, "toast": 1833, "passengers": 1834, "hide": 1835, "yarmouth": 1836, "southend": 1837, "pier": 1838, "recollect": 1839, "shaking": 1840, "weren": 1841, "brightening": 1842, "disgraceful": 1843, "rises": 1844, "winter": 1845, "alike": 1846, "cake": 1847, "price": 1848, "smoke": 1849, "rat": 1850, "ii": 1851, "compromise": 1852, "fears": 1853, "hearts": 1854, "ghostly": 1855, "creep": 1856, "pitched": 1857, "margin": 1858, "moon": 1859, "kiss": 1860, "throws": 1861, "strangely": 1862, "centuries": 1863, "mankind": 1864, "bet": 1865, "corpses": 1866, "drinking": 1867, "lug": 1868, "soaked": 1869, "helping": 1870, "spoils": 1871, "idiot": 1872, "mutter": 1873, "exclaim": 1874, "breath": 1875, "sleeve": 1876, "cursing": 1877, "diet": 1878, "quantity": 1879, "grasp": 1880, "express": 1881, "usual": 1882, "kicking": 1883, "rocks": 1884, "struggles": 1885, "colds": 1886, "swear": 1887, "pub": 1888, "hailed": 1889, "trifle": 1890, "shape": 1891, "paid": 1892, "growling": 1893, "female": 1894, "murderer": 1895, "backing": 1896, "observed": 1897, "satisfaction": 1898, "discuss": 1899, "begun": 1900, "proposed": 1901, "saying": 1902, "elderly": 1903, "puts": 1904, "provisions": 1905, "kitchen": 1906, "charwoman": 1907, "injured": 1908, "knees": 1909, "mark": 1910, "measure": 1911, "arrive": 1912, "force": 1913, "foolish": 1914, "swell": 1915, "manage": 1916, "windy": 1917, "lightly": 1918, "glittering": 1919, "easier": 1920, "simpler": 1921, "funeral": 1922, "comb": 1923, "shaving": 1924, "sounds": 1925, "towels": 1926, "religiously": 1927, "pick": 1928, "sand": 1929, "insulting": 1930, "swim": 1931, "urged": 1932, "shilling": 1933, "socks": 1934, "leather": 1935, "atmosphere": 1936, "cussedness": 1937, "behaviour": 1938, "escape": 1939, "oath": 1940, "affair": 1941, "bell": 1942, "dashed": 1943, "sniff": 1944, "pleasantly": 1945, "forced": 1946, "umbrellas": 1947, "neat": 1948, "drunk": 1949, "difference": 1950, "honour": 1951, "widow": 1952, "decline": 1953, "fifteen": 1954, "cost": 1955, "complained": 1956, "beach": 1957, "fruit": 1958, "mouch": 1959, "cooking": 1960, "surprises": 1961, "readiness": 1962, "uncanny": 1963, "intended": 1964, "irritate": 1965, "pockets": 1966, "energetic": 1967, "strapped": 1968, "occurred": 1969, "misery": 1970, "5": 1971, "p": 1972, "cups": 1973, "c": 1974, "stared": 1975, "sworn": 1976, "pretended": 1977, "encourage": 1978, "tossed": 1979, "gather": 1980, "drive": 1981, "concerning": 1982, "retorted": 1983, "lucky": 1984, "strain": 1985, "reminded": 1986, "existence": 1987, "wide": 1988, "fat": 1989, "thunder": 1990, "dressing": 1991, "absurd": 1992, "doorstep": 1993, "occasional": 1994, "precisely": 1995, "completely": 1996, "heat": 1997, "landlady": 1998, "sign": 1999, "shelter": 2000, "tapped": 2001, "jumped": 2002, "higher": 2003, "poem": 2004, "machine": 2005, "ones": 2006, "tap": 2007, "correct": 2008, "curdling": 2009, "roll": 2010, "abandoned": 2011, "murder": 2012, "straw": 2013, "elder": 2014, "probably": 2015, "corpse": 2016, "drove": 2017, "master": 2018, "convinced": 2019, "32": 2020, "prow": 2021, "stivvings": 2022, "flashing": 2023, "drifting": 2024, "trim": 2025, "elizabeth": 2026, "signs": 2027, "chucked": 2028, "steal": 2029, "beloved": 2030, "brutal": 2031, "drag": 2032, "loud": 2033, "clamour": 2034, "latticed": 2035, "oaths": 2036, "staircase": 2037, "market": 2038, "thoughtless": 2039, "recovering": 2040, "thereupon": 2041, "panelled": 2042, "carving": 2043, "wooden": 2044, "merton": 2045, "parents": 2046, "yearned": 2047, "ideas": 2048, "terribly": 2049, "singularly": 2050, "ornaments": 2051, "flower": 2052, "bedroom": 2053, "delicate": 2054, "spots": 2055, "painfully": 2056, "admire": 2057, "considered": 2058, "weight": 2059, "escaped": 2060, "charming": 2061, "growing": 2062, "peeping": 2063, "busy": 2064, "towns": 2065, "alive": 2066, "helpless": 2067, "charged": 2068, "joke": 2069, "procession": 2070, "insisted": 2071, "penny": 2072, "entrance": 2073, "trailed": 2074, "regular": 2075, "tricks": 2076, "moulsey": 2077, "tangle": 2078, "caps": 2079, "coloured": 2080, "flowers": 2081, "dotted": 2082, "decked": 2083, "inhabitants": 2084, "sails": 2085, "landscape": 2086, "keeps": 2087, "showed": 2088, "utterly": 2089, "lace": 2090, "ridiculous": 2091, "assured": 2092, "cushion": 2093, "sighed": 2094, "christian": 2095, "splash": 2096, "paused": 2097, "accomplished": 2098, "lips": 2099, "involuntary": 2100, "murmur": 2101, "dusty": 2102, "bolt": 2103, "upright": 2104, "grasped": 2105, "dense": 2106, "lane": 2107, "tall": 2108, "hills": 2109, "dignity": 2110, "repeated": 2111, "tombstone": 2112, "stranger": 2113, "inch": 2114, "shepperton": 2115, "concentrated": 2116, "jar": 2117, "sticking": 2118, "dared": 2119, "thanked": 2120, "hung": 2121, "declined": 2122, "chuck": 2123, "riverside": 2124, "lazy": 2125, "backwaters": 2126, "chains": 2127, "instinct": 2128, "tear": 2129, "allowed": 2130, "fetch": 2131, "accompaniment": 2132, "addressing": 2133, "performance": 2134, "argument": 2135, "requests": 2136, "seizing": 2137, "explanation": 2138, "appearing": 2139, "nono": 2140, "cleaned": 2141, "dash": 2142, "annoying": 2143, "honestly": 2144, "restless": 2145, "slow": 2146, "graceful": 2147, "spoil": 2148, "amusing": 2149, "professor": 2150, "intense": 2151, "earnest": 2152, "annoyance": 2153, "indignation": 2154, "agony": 2155, "insulted": 2156, "attempt": 2157, "foot": 2158, "idiots": 2159, "grinding": 2160, "thanks": 2161, "works": 2162, "cromwell": 2163, "immense": 2164, "handle": 2165, "towers": 2166, "haunted": 2167, "disentangle": 2168, "dances": 2169, "knowledge": 2170, "witnessed": 2171, "example": 2172, "disturbed": 2173, "necessary": 2174, "flashed": 2175, "attitudes": 2176, "shouting": 2177, "hid": 2178, "butt": 2179, "necks": 2180, "breathless": 2181, "turns": 2182, "ease": 2183, "jump": 2184, "dreaming": 2185, "strains": 2186, "heavenly": 2187, "melody": 2188, "contained": 2189, "tight": 2190, "appeal": 2191, "magna": 2192, "charta": 2193, "tiny": 2194, "arches": 2195, "tale": 2196, "drown": 2197, "receive": 2198, "boil": 2199, "squatted": 2200, "intellect": 2201, "animal": 2202, "splutter": 2203, "handy": 2204, "undress": 2205, "branches": 2206, "swallowed": 2207, "shadowy": 2208, "wondrous": 2209, "sorrow": 2210, "rode": 2211, "gloom": 2212, "paths": 2213, "awake": 2214, "shame": 2215, "shops": 2216, "listened": 2217, "chess": 2218, "stroll": 2219, "retiring": 2220, "branch": 2221, "blowing": 2222, "louder": 2223, "mistaken": 2224, "amused": 2225, "careful": 2226, "dish": 2227, "doors": 2228, "groups": 2229, "troops": 2230, "mercenaries": 2231, "group": 2232, "ranks": 2233, "favour": 2234, "patient": 2235, "dust": 2236, "midst": 2237, "hurried": 2238, "desperate": 2239, "rested": 2240, "cottage": 2241, "grounds": 2242, "albans": 2243, "garden": 2244, "guilty": 2245, "greatest": 2246, "billiard": 2247, "recommend": 2248, "rooms": 2249, "emotion": 2250, "spoon": 2251, "shows": 2252, "mocking": 2253, "rowed": 2254, "heroine": 2255, "novel": 2256, "fairy": 2257, "sweetest": 2258, "bearing": 2259, "moored": 2260, "chairs": 2261, "lake": 2262, "imposing": 2263, "dating": 2264, "nestling": 2265, "fraternity": 2266, "founded": 2267, "type": 2268, "chatting": 2269, "dignified": 2270, "final": 2271, "stolid": 2272, "gravy": 2273, "learning": 2274, "enjoying": 2275, "awkward": 2276, "peel": 2277, "peeling": 2278, "hackneyed": 2279, "attitude": 2280, "teach": 2281, "amount": 2282, "jefferson": 2283, "company": 2284, "cheery": 2285, "becoming": 2286, "toddy": 2287, "recollection": 2288, "generation": 2289, "rafting": 2290, "difficulty": 2291, "youngsters": 2292, "novices": 2293, "pond": 2294, "extremely": 2295, "kew": 2296, "outrigger": 2297, "cox": 2298, "oar": 2299, "clinging": 2300, "heartily": 2301, "ethelred": 2302, "smiling": 2303, "perch": 2304, "rod": 2305, "per": 2306, "cent": 2307, "parlour": 2308, "weighed": 2309, "pieces": 2310, "ours": 2311, "photo": 2312, "photograph": 2313, "w": 2314, "pictures": 2315, "relic": 2316, "coffin": 2317, "gipsy": 2318, "sufferings": 2319, "maladies": 2320, "useful": 2321, "complaint": 2322, "agree": 2323, "suggests": 2324, "giddiness": 2325, "patent": 2326, "pill": 2327, "circular": 2328, "advertisement": 2329, "plunged": 2330, "awhile": 2331, "pages": 2332, "modified": 2333, "cholera": 2334, "conscientiously": 2335, "letters": 2336, "malady": 2337, "prevailed": 2338, "reflected": 2339, "selfish": 2340, "zymosis": 2341, "pondered": 2342, "sudden": 2343, "tip": 2344, "scarlet": 2345, "healthy": 2346, "chum": 2347, "discover": 2348, "clutched": 2349, "cowardly": 2350, "prescription": 2351, "nearest": 2352, "6": 2353, "earliest": 2354, "laziness": 2355, "pills": 2356, "clumps": 2357, "fashioned": 2358, "describing": 2359, "steak": 2360, "meand": 2361, "mental": 2362, "drowsy": 2363, "biggest": 2364, "wednesday": 2365, "thursday": 2366, "sell": 2367, "youth": 2368, "advised": 2369, "pressing": 2370, "lifetime": 2371, "voyage": 2372, "recommended": 2373, "latter": 2374, "cheaper": 2375, "discontented": 2376, "uppish": 2377, "chicken": 2378, "queer": 2379, "advise": 2380, "anecdotes": 2381, "channel": 2382, "mild": 2383, "query": 2384, "confess": 2385, "puzzled": 2386, "pickles": 2387, "tasted": 2388, "excellent": 2389, "sickness": 2390, "heaves": 2391, "touches": 2392, "tendency": 2393, "greatly": 2394, "suited": 2395, "foolishness": 2396, "lest": 2397, "subsequently": 2398, "dismissed": 2399, "memory": 2400, "birds": 2401, "harsh": 2402, "awed": 2403, "hush": 2404, "breathes": 2405, "noiseless": 2406, "guard": 2407, "sombre": 2408, "eaten": 2409, "lighted": 2410, "nestled": 2411, "loves": 2412, "whispering": 2413, "ashes": 2414, "burnt": 2415, "lulled": 2416, "lapping": 2417, "rustling": 2418, "fret": 2419, "bygone": 2420, "painted": 2421, "lured": 2422, "stately": 2423, "rouse": 2424, "yearning": 2425, "unattainable": 2426, "raw": 2427, "chop": 2428, "seaweed": 2429, "chill": 2430, "supposing": 2431, "greet": 2432, "hint": 2433, "tumbles": 2434, "task": 2435, "wishing": 2436, "starts": 2437, "meanwhile": 2438, "rainwater": 2439, "exceedingly": 2440, "salt": 2441, "soup": 2442, "tobacco": 2443, "cheers": 2444, "elephant": 2445, "exploded": 2446, "terrible": 2447, "cries": 2448, "dearly": 2449, "frantically": 2450, "yelling": 2451, "lustily": 2452, "dimly": 2453, "observe": 2454, "dawn": 2455, "muddy": 2456, "unnecessarily": 2457, "belief": 2458, "owing": 2459, "quarrelsome": 2460, "whispers": 2461, "folks": 2462, "solitude": 2463, "pious": 2464, "expense": 2465, "fourteen": 2466, "fights": 2467, "inspection": 2468, "collect": 2469, "lead": 2470, "argue": 2471, "adjourned": 2472, "remark": 2473, "assembled": 2474, "settle": 2475, "catalogue": 2476, "pencil": 2477, "commotion": 2478, "frame": 2479, "maker": 2480, "worry": 2481, "yourselves": 2482, "nails": 2483, "regards": 2484, "hopes": 2485, "lend": 2486, "tools": 2487, "dance": 2488, "hinder": 2489, "tying": 2490, "finger": 2491, "fourth": 2492, "grunt": 2493, "heavens": 2494, "beside": 2495, "fools": 2496, "notes": 2497, "smash": 2498, "spend": 2499, "picking": 2500, "admiring": 2501, "plaster": 2502, "heavily": 2503, "discarded": 2504, "upper": 2505, "tore": 2506, "expensive": 2507, "cloy": 2508, "bleed": 2509, "aching": 2510, "oars": 2511, "freedom": 2512, "dreamy": 2513, "er": 2514, "shallows": 2515, "lilies": 2516, "thirst": 2517, "liable": 2518, "draws": 2519, "adopted": 2520, "cosy": 2521, "stuffy": 2522, "basin": 2523, "tackle": 2524, "gigantic": 2525, "dip": 2526, "complexion": 2527, "virtue": 2528, "specially": 2529, "cutting": 2530, "east": 2531, "hop": 2532, "catches": 2533, "carries": 2534, "ocean": 2535, "strike": 2536, "retires": 2537, "swimming": 2538, "pretend": 2539, "plunge": 2540, "ordinarily": 2541, "withdrew": 2542, "opposition": 2543, "suits": 2544, "flannels": 2545, "influence": 2546, "learn": 2547, "impostor": 2548, "impressed": 2549, "handkerchiefs": 2550, "wipe": 2551, "advantages": 2552, "brushes": 2553, "indigestible": 2554, "raised": 2555, "laden": 2556, "flavour": 2557, "sausage": 2558, "splendid": 2559, "mellow": 2560, "power": 2561, "carry": 2562, "winded": 2563, "referred": 2564, "steed": 2565, "driver": 2566, "notwithstanding": 2567, "oppressive": 2568, "parcels": 2569, "crewe": 2570, "waved": 2571, "brandy": 2572, "smelt": 2573, "worst": 2574, "directed": 2575, "sovereign": 2576, "consider": 2577, "orphan": 2578, "eloquent": 2579, "terms": 2580, "instinctively": 2581, "argued": 2582, "sixpence": 2583, "means": 2584, "canal": 2585, "parish": 2586, "waking": 2587, "gained": 2588, "visitors": 2589, "shan": 2590, "tomatoes": 2591, "sleepy": 2592, "utensils": 2593, "moved": 2594, "cigar": 2595, "potter": 2596, "irritated": 2597, "yawned": 2598, "senseless": 2599, "unpack": 2600, "mortal": 2601, "remained": 2602, "comment": 2603, "kettles": 2604, "bottles": 2605, "cakes": 2606, "tomato": 2607, "teaspoon": 2608, "trod": 2609, "smashed": 2610, "slipper": 2611, "scrape": 2612, "ambition": 2613, "curse": 2614, "aim": 2615, "accomplishing": 2616, "placed": 2617, "tumble": 2618, "worldly": 2619, "afloat": 2620, "lain": 2621, "priceless": 2622, "hideous": 2623, "dispute": 2624, "awoke": 2625, "chunk": 2626, "shrieked": 2627, "remembered": 2628, "cared": 2629, "vulgar": 2630, "chops": 2631, "newspaper": 2632, "showers": 2633, "thunderstorms": 2634, "cloud": 2635, "stirred": 2636, "bitterly": 2637, "barometer": 2638, "misleading": 2639, "hanging": 2640, "pointing": 2641, "morrow": 2642, "peg": 2643, "drought": 2644, "sunstroke": 2645, "content": 2646, "foretold": 2647, "falls": 2648, "ely": 2649, "circumstances": 2650, "continuing": 2651, "angry": 2652, "vague": 2653, "europe": 2654, "wasting": 2655, "sneaked": 2656, "grapes": 2657, "japanese": 2658, "greengrocer": 2659, "services": 2660, "crops": 2661, "crime": 2662, "assisted": 2663, "21": 2664, "prove": 2665, "alibi": 2666, "dawned": 2667, "sensitive": 2668, "superintendent": 2669, "atlantic": 2670, "collected": 2671, "amidst": 2672, "rumour": 2673, "loop": 2674, "confident": 2675, "begged": 2676, "gimme": 2677, "wended": 2678, "deeply": 2679, "instructive": 2680, "leaf": 2681, "blushing": 2682, "deeper": 2683, "trembling": 2684, "brink": 2685, "glinting": 2686, "towpath": 2687, "glimpses": 2688, "tudors": 2689, "dreamily": 2690, "musing": 2691, "mused": 2692, "kyningestun": 2693, "crossed": 2694, "legions": 2695, "attractions": 2696, "patronised": 2697, "entered": 2698, "edwy": 2699, "feast": 2700, "boar": 2701, "revelry": 2702, "bursts": 2703, "din": 2704, "faced": 2705, "drunken": 2706, "crash": 2707, "kings": 2708, "rise": 2709, "royal": 2710, "strained": 2711, "cloaked": 2712, "steel": 2713, "prancing": 2714, "silks": 2715, "complicated": 2716, "bricks": 2717, "creak": 2718, "buy": 2719, "shopman": 2720, "staggered": 2721, "hero": 2722, "doubtless": 2723, "curiosity": 2724, "maniac": 2725, "depressing": 2726, "prices": 2727, "single": 2728, "couples": 2729, "notions": 2730, "harmless": 2731, "unborn": 2732, "bronchitis": 2733, "rheumatic": 2734, "fog": 2735, "false": 2736, "suffered": 2737, "sob": 2738, "sake": 2739, "excuse": 2740, "kinds": 2741, "term": 2742, "baked": 2743, "treasures": 2744, "mugs": 2745, "household": 2746, "pink": 2747, "gush": 2748, "future": 2749, "sarah": 2750, "heartedness": 2751, "dusted": 2752, "erect": 2753, "verge": 2754, "irritates": 2755, "jeer": 2756, "excuses": 2757, "circumstance": 2758, "probable": 2759, "depth": 2760, "familiar": 2761, "loveliness": 2762, "daughter": 2763, "tapestry": 2764, "presents": 2765, "margate": 2766, "howled": 2767, "lose": 2768, "repeat": 2769, "moss": 2770, "sober": 2771, "ivy": 2772, "clustering": 2773, "rang": 2774, "cities": 2775, "sides": 2776, "lonesome": 2777, "answering": 2778, "studied": 2779, "absorbed": 2780, "courage": 2781, "swore": 2782, "expressed": 2783, "advisability": 2784, "pretending": 2785, "treat": 2786, "mob": 2787, "curl": 2788, "extent": 2789, "crazy": 2790, "whirl": 2791, "hedge": 2792, "performs": 2793, "boulter": 2794, "parasols": 2795, "streaming": 2796, "ribbons": 2797, "shade": 2798, "rainbow": 2799, "colours": 2800, "belt": 2801, "wise": 2802, "yellows": 2803, "respect": 2804, "attract": 2805, "costumes": 2806, "rubbed": 2807, "occasionally": 2808, "returning": 2809, "smooth": 2810, "firm": 2811, "touched": 2812, "visibly": 2813, "fitful": 2814, "splashed": 2815, "changed": 2816, "sigh": 2817, "brightened": 2818, "chap": 2819, "dashing": 2820, "pint": 2821, "trunks": 2822, "fortunately": 2823, "chilly": 2824, "wheezy": 2825, "epitaphs": 2826, "assume": 2827, "inscriptions": 2828, "lack": 2829, "drank": 2830, "porch": 2831, "thatched": 2832, "cottages": 2833, "hollow": 2834, "sinful": 2835, "forgave": 2836, "bald": 2837, "spry": 2838, "bewildered": 2839, "parts": 2840, "roused": 2841, "capable": 2842, "figure": 2843, "memorial": 2844, "whispered": 2845, "calling": 2846, "sits": 2847, "trick": 2848, "served": 2849, "gallon": 2850, "mixing": 2851, "cause": 2852, "holding": 2853, "blackmailing": 2854, "pursue": 2855, "shameful": 2856, "willows": 2857, "trespassing": 2858, "enable": 2859, "dissatisfied": 2860, "disposition": 2861, "belonged": 2862, "society": 2863, "gruffly": 2864, "roughs": 2865, "address": 2866, "summon": 2867, "timid": 2868, "imposition": 2869, "owners": 2870, "riparian": 2871, "streams": 2872, "mentioned": 2873, "caused": 2874, "instincts": 2875, "justice": 2876, "implies": 2877, "hostess": 2878, "generous": 2879, "conservatory": 2880, "bars": 2881, "easing": 2882, "snigger": 2883, "murmurs": 2884, "delight": 2885, "commences": 2886, "finds": 2887, "admiral": 2888, "opening": 2889, "contest": 2890, "jove": 2891, "suggesting": 2892, "approaching": 2893, "earthquake": 2894, "swept": 2895, "polished": 2896, "ruler": 2897, "navee": 2898, "inner": 2899, "recorded": 2900, "highly": 2901, "proceedings": 2902, "pathetic": 2903, "arrived": 2904, "funniest": 2905, "written": 2906, "tragedy": 2907, "pathos": 2908, "fetched": 2909, "pleased": 2910, "suggest": 2911, "progressed": 2912, "slightest": 2913, "shriek": 2914, "popular": 2915, "effective": 2916, "mountains": 2917, "sobbed": 2918, "softly": 2919, "servant": 2920, "sweetly": 2921, "weirs": 2922, "bent": 2923, "swing": 2924, "injuring": 2925, "sculled": 2926, "corners": 2927, "ugly": 2928, "charles": 2929, "likewise": 2930, "causes": 2931, "oatlands": 2932, "stole": 2933, "grotto": 2934, "special": 2935, "dare": 2936, "stakes": 2937, "cassivelaunus": 2938, "choke": 2939, "longing": 2940, "furious": 2941, "annoyed": 2942, "rage": 2943, "ungrateful": 2944, "disappearance": 2945, "haste": 2946, "missing": 2947, "worrying": 2948, "twisted": 2949, "cautiously": 2950, "dummy": 2951, "lays": 2952, "wound": 2953, "rests": 2954, "entangled": 2955, "rope": 2956, "boveney": 2957, "helplessly": 2958, "rapidly": 2959, "calls": 2960, "confound": 2961, "tidied": 2962, "maiden": 2963, "sympathy": 2964, "noticing": 2965, "scattered": 2966, "lighten": 2967, "occupied": 2968, "experiences": 2969, "frock": 2970, "aground": 2971, "nicely": 2972, "shawl": 2973, "mary": 2974, "cow": 2975, "penton": 2976, "trudged": 2977, "seriously": 2978, "benson": 2979, "cleeve": 2980, "youyou": 2981, "offend": 2982, "ghost": 2983, "judgment": 2984, "fast": 2985, "clearly": 2986, "recollected": 2987, "lure": 2988, "concertina": 2989, "sounded": 2990, "harmony": 2991, "performed": 2992, "lovable": 2993, "lor": 2994, "gratitude": 2995, "soldiers": 2996, "overcome": 2997, "appointed": 2998, "drained": 2999, "valley": 3000, "coal": 3001, "satisfied": 3002, "fitted": 3003, "demons": 3004, "struggled": 3005, "hinges": 3006, "endeavouring": 3007, "persuade": 3008, "unrolled": 3009, "process": 3010, "violently": 3011, "guessed": 3012, "cuckoo": 3013, "lantern": 3014, "contentment": 3015, "exhibited": 3016, "obtained": 3017, "forgiving": 3018, "substantial": 3019, "soulless": 3020, "ray": 3021, "grin": 3022, "wit": 3023, "veriest": 3024, "reign": 3025, "tempered": 3026, "beamed": 3027, "desires": 3028, "unpleasant": 3029, "sized": 3030, "longed": 3031, "separate": 3032, "bumps": 3033, "disappeared": 3034, "gimlet": 3035, "unkind": 3036, "accumulate": 3037, "mighty": 3038, "vast": 3039, "ears": 3040, "awe": 3041, "strayed": 3042, "temple": 3043, "taught": 3044, "stained": 3045, "cheek": 3046, "angels": 3047, "tangled": 3048, "comrades": 3049, "stayed": 3050, "burned": 3051, "questioned": 3052, "bleeding": 3053, "lo": 3054, "whereof": 3055, "determination": 3056, "utter": 3057, "bargain": 3058, "hauled": 3059, "flung": 3060, "sprang": 3061, "unbolted": 3062, "foggy": 3063, "cabbages": 3064, "counted": 3065, "obliged": 3066, "advice": 3067, "game": 3068, "policemen": 3069, "hiding": 3070, "relish": 3071, "spluttered": 3072, "missed": 3073, "whereupon": 3074, "roaring": 3075, "shrieks": 3076, "mouths": 3077, "preventing": 3078, "chivied": 3079, "fork": 3080, "dancing": 3081, "operations": 3082, "june": 3083, "yeomen": 3084, "cloth": 3085, "meaning": 3086, "stir": 3087, "duncroft": 3088, "armed": 3089, "clatter": 3090, "squires": 3091, "sword": 3092, "forms": 3093, "vacant": 3094, "peering": 3095, "lights": 3096, "crouching": 3097, "height": 3098, "thame": 3099, "bass": 3100, "score": 3101, "lazily": 3102, "war": 3103, "bustle": 3104, "rowers": 3105, "charter": 3106, "signing": 3107, "slippery": 3108, "grip": 3109, "slid": 3110, "vain": 3111, "larger": 3112, "ride": 3113, "fierce": 3114, "drops": 3115, "runningmede": 3116, "boleyn": 3117, "fishers": 3118, "cursed": 3119, "signed": 3120, "personal": 3121, "priory": 3122, "hever": 3123, "courting": 3124, "drat": 3125, "pirates": 3126, "bells": 3127, "godwin": 3128, "stretches": 3129, "august": 3130, "goodish": 3131, "direct": 3132, "realise": 3133, "meek": 3134, "suggestions": 3135, "scorn": 3136, "nooks": 3137, "beershop": 3138, "27": 3139, "requested": 3140, "disguise": 3141, "peculiarly": 3142, "suggestive": 3143, "occupants": 3144, "letting": 3145, "bare": 3146, "monkey": 3147, "daresay": 3148, "proportion": 3149, "kicked": 3150, "pine": 3151, "juice": 3152, "scissors": 3153, "stretchers": 3154, "stirring": 3155, "battered": 3156, "sank": 3157, "hurled": 3158, "ballet": 3159, "grumbled": 3160, "dreams": 3161, "mystic": 3162, "glory": 3163, "sky": 3164, "wrapping": 3165, "realm": 3166, "cursory": 3167, "grateful": 3168, "shocked": 3169, "grieved": 3170, "departure": 3171, "shattered": 3172, "warwick": 3173, "quarry": 3174, "meadows": 3175, "narrow": 3176, "memories": 3177, "templars": 3178, "contains": 3179, "chamber": 3180, "secret": 3181, "hurley": 3182, "club": 3183, "member": 3184, "prayer": 3185, "hambledon": 3186, "vigorous": 3187, "terriers": 3188, "appreciable": 3189, "haymarket": 3190, "yorkshire": 3191, "judging": 3192, "yelp": 3193, "experiment": 3194, "collie": 3195, "curiously": 3196, "uninterrupted": 3197, "equally": 3198, "tyke": 3199, "kissed": 3200, "trot": 3201, "enemy": 3202, "prey": 3203, "assassin": 3204, "follows": 3205, "unhealthy": 3206, "vegetables": 3207, "adopting": 3208, "principle": 3209, "collection": 3210, "baskets": 3211, "hate": 3212, "yearn": 3213, "drift": 3214, "whistling": 3215, "boiler": 3216, "assistance": 3217, "thank": 3218, "accustomed": 3219, "containing": 3220, "owed": 3221, "winning": 3222, "sluggish": 3223, "emptied": 3224, "escaping": 3225, "meadow": 3226, "climb": 3227, "damaged": 3228, "poet": 3229, "sarcastic": 3230, "discouragement": 3231, "amateur": 3232, "strangeness": 3233, "leslie": 3234, "hodgson": 3235, "imagined": 3236, "hushed": 3237, "gossip": 3238, "passages": 3239, "odds": 3240, "fascinating": 3241, "require": 3242, "scraping": 3243, "overhauled": 3244, "fished": 3245, "assist": 3246, "helped": 3247, "precedent": 3248, "safe": 3249, "nutritious": 3250, "constitutional": 3251, "howling": 3252, "injure": 3253, "sale": 3254, "acquire": 3255, "members": 3256, "active": 3257, "practise": 3258, "describe": 3259, "visitor": 3260, "caution": 3261, "listening": 3262, "grunted": 3263, "worm": 3264, "space": 3265, "skiplake": 3266, "thoughtfully": 3267, "overdo": 3268, "bark": 3269, "defeated": 3270, "routed": 3271, "partook": 3272, "share": 3273, "itat": 3274, "worries": 3275, "feats": 3276, "appealed": 3277, "adds": 3278, "reaching": 3279, "exaggerate": 3280, "proud": 3281, "believed": 3282, "stories": 3283, "constructed": 3284, "raft": 3285, "devoted": 3286, "lea": 3287, "smart": 3288, "sixteen": 3289, "proceeded": 3290, "detail": 3291, "lesson": 3292, "flash": 3293, "engaged": 3294, "attracted": 3295, "jacket": 3296, "jokes": 3297, "belonging": 3298, "released": 3299, "reef": 3300, "upside": 3301, "refused": 3302, "hector": 3303, "wetted": 3304, "ended": 3305, "considerable": 3306, "linger": 3307, "wessex": 3308, "alfred": 3309, "parliament": 3310, "plague": 3311, "1625": 3312, "parliamentary": 3313, "continually": 3314, "mapledurham": 3315, "surely": 3316, "aged": 3317, "deceived": 3318, "monotony": 3319, "unheeded": 3320, "formed": 3321, "contradict": 3322, "statement": 3323, "angling": 3324, "conscientious": 3325, "fishy": 3326, "failure": 3327, "pike": 3328, "dace": 3329, "gudgeon": 3330, "pause": 3331, "sturgeon": 3332, "doubled": 3333, "opposed": 3334, "wur": 3335, "carrier": 3336, "culham": 3337, "photographer": 3338, "surrounding": 3339, "copies": 3340, "fortifications": 3341, "romans": 3342, "barley": 3343, "mow": 3344, "modern": 3345, "lee": 3346, "nuneham": 3347, "obelisk": 3348, "mill": 3349, "aspects": 3350, "mournful": 3351, "constitutionally": 3352, "hire": 3353, "fossil": 3354, "whale": 3355, "sunshine": 3356, "paddle": 3357, "enjoyment": 3358, "alhambra": 3359, "restaurant": 3360, "napkins": 3361, "invalids": 3362, "fatal": 3363, "prescriptions": 3364, "cure": 3365, "overworked": 3366, "rolling": 3367, "lodges": 3368, "usgeorge": 3369, "samuel": 3370, "werebad": 3371, "seedy": 3372, "detailed": 3373, "medicine": 3374, "impelled": 3375, "dealt": 3376, "virulent": 3377, "diagnosis": 3378, "correspond": 3379, "sensations": 3380, "museum": 3381, "treatment": 3382, "ailment": 3383, "touchhay": 3384, "unthinking": 3385, "idly": 3386, "indolently": 3387, "distemper": 3388, "intosome": 3389, "devastating": 3390, "scourge": 3391, "knowand": 3392, "glanced": 3393, "premonitory": 3394, "frozen": 3395, "horror": 3396, "listlessness": 3397, "despair": 3398, "feverread": 3399, "symptomsdiscovered": 3400, "itwondered": 3401, "vitus": 3402, "dancefound": 3403, "sift": 3404, "alphabeticallyread": 3405, "ague": 3406, "sickening": 3407, "acute": 3408, "relieved": 3409, "complications": 3410, "diphtheria": 3411, "plodded": 3412, "conclude": 3413, "invidious": 3414, "reservation": 3415, "pharmacology": 3416, "gout": 3417, "malignant": 3418, "aware": 3419, "boyhood": 3420, "acquisition": 3421, "hospitals": 3422, "hospital": 3423, "diploma": 3424, "timed": 3425, "induced": 3426, "patted": 3427, "waist": 3428, "crawled": 3429, "decrepit": 3430, "talks": 3431, "seventeen": 3432, "patients": 3433, "wrist": 3434, "expecting": 3435, "ita": 3436, "itand": 3437, "butted": 3438, "co": 3439, "operative": 3440, "oblige": 3441, "lb": 3442, "pt": 3443, "resultspeaking": 3444, "myselfthat": 3445, "preserved": 3446, "disinclination": 3447, "suffer": 3448, "infancy": 3449, "martyr": 3450, "science": 3451, "skulking": 3452, "devil": 3453, "cured": 3454, "mefor": 3455, "clump": 3456, "loss": 3457, "sothose": 3458, "remedies": 3459, "efficacious": 3460, "dispensary": 3461, "hearth": 3462, "acting": 3463, "illustrative": 3464, "fancies": 3465, "sadly": 3466, "check": 3467, "tray": 3468, "toyed": 3469, "rhubarb": 3470, "foodan": 3471, "refilled": 3472, "resumed": 3473, "unanimous": 3474, "itwhatever": 3475, "washad": 3476, "overwork": 3477, "overstrain": 3478, "brains": 3479, "restore": 3480, "equilibrium": 3481, "student": 3482, "physicianary": 3483, "seek": 3484, "retired": 3485, "madding": 3486, "lanessome": 3487, "hidden": 3488, "fairies": 3489, "worldsome": 3490, "perched": 3491, "eyrie": 3492, "cliffs": 3493, "whence": 3494, "surging": 3495, "waves": 3496, "humpy": 3497, "referee": 3498, "baccy": 3499, "implanted": 3500, "airy": 3501, "adieu": 3502, "swagger": 3503, "francis": 3504, "drake": 3505, "christopher": 3506, "columbus": 3507, "wan": 3508, "solid": 3509, "ashore": 3510, "thoroughly": 3511, "benefit": 3512, "berth": 3513, "reduction": 3514, "eighteenpence": 3515, "bilious": 3516, "affectionately": 3517, "somersaults": 3518, "himselfmy": 3519, "lawcame": 3520, "north": 3521, "series": 3522, "grill": 3523, "consisted": 3524, "courses": 3525, "sixsoup": 3526, "entree": 3527, "joint": 3528, "poultry": 3529, "salad": 3530, "sweets": 3531, "dessert": 3532, "eater": 3533, "sheerness": 3534, "eitherseemed": 3535, "announcement": 3536, "aroused": 3537, "ham": 3538, "fried": 3539, "greens": 3540, "propped": 3541, "leeward": 3542, "soda": 3543, "gorging": 3544, "broth": 3545, "steamed": 3546, "regretfully": 3547, "belongs": 3548, "seasaid": 3549, "affectationsaid": 3550, "berths": 3551, "souls": 3552, "mate": 3553, "sickon": 3554, "loads": 3555, "sailors": 3556, "swarm": 3557, "seeming": 3558, "enigma": 3559, "port": 3560, "holes": 3561, "shoulder": 3562, "voyages": 3563, "explaining": 3564, "sailor": 3565, "envious": 3566, "cape": 3567, "horn": 3568, "vessel": 3569, "wrecked": 3570, "shaky": 3571, "ahyes": 3572, "preventive": 3573, "balancing": 3574, "pitches": 3575, "backwards": 3576, "balance": 3577, "constant": 3578, "occupy": 3579, "sleepier": 3580, "sixpenny": 3581, "includes": 3582, "ad": 3583, "lib": 3584, "imply": 3585, "slop": 3586, "ditto": 3587, "impressions": 3588, "groundless": 3589, "adjourns": 3590, "maps": 3591, "chertsey": 3592, "saturdays": 3593, "patriarchal": 3594, "fades": 3595, "sorrowing": 3596, "ceased": 3597, "moorhen": 3598, "plaintive": 3599, "croak": 3600, "corncrake": 3601, "stirs": 3602, "couch": 3603, "army": 3604, "tread": 3605, "chase": 3606, "lingering": 3607, "unseen": 3608, "sighing": 3609, "throne": 3610, "folds": 3611, "darkening": 3612, "phantom": 3613, "pale": 3614, "reigns": 3615, "frugal": 3616, "cooked": 3617, "undertone": 3618, "pauses": 3619, "prattles": 3620, "secrets": 3621, "yearswill": 3622, "olda": 3623, "changing": 3624, "yielding": 3625, "stoops": 3626, "clingingly": 3627, "flows": 3628, "seatill": 3629, "outtill": 3630, "everyday": 3631, "speaktill": 3632, "againyoung": 3633, "furrowed": 3634, "sins": 3635, "follies": 3636, "heartsweet": 3637, "nursed": 3638, "breastere": 3639, "wiles": 3640, "civilization": 3641, "poisoned": 3642, "sneers": 3643, "artificiality": 3644, "rained": 3645, "poetry": 3646, "harrisno": 3647, "weeps": 3648, "fill": 3649, "worcester": 3650, "mermaid": 3651, "hark": 3652, "mermaids": 3653, "spirits": 3654, "chanting": 3655, "dirges": 3656, "finest": 3657, "tastedput": 3658, "paradise": 3659, "nectar": 3660, "timely": 3661, "rainy": 3662, "puddly": 3663, "proceed": 3664, "flops": 3665, "clings": 3666, "herculean": 3667, "hoist": 3668, "retorts": 3669, "leggo": 3670, "yells": 3671, "pulls": 3672, "pegs": 3673, "mallet": 3674, "baling": 3675, "spilled": 3676, "thundering": 3677, "blazes": 3678, "blarmed": 3679, "hopeless": 3680, "attempting": 3681, "thirds": 3682, "luckily": 3683, "inebriates": 3684, "restores": 3685, "induce": 3686, "volcano": 3687, "seathe": 3688, "peacefully": 3689, "thieves": 3690, "murderers": 3691, "determining": 3692, "hitting": 3693, "ruffian": 3694, "preparing": 3695, "recognising": 3696, "rubbing": 3697, "blown": 3698, "raise": 3699, "muffled": 3700, "ruin": 3701, "trampled": 3702, "aggressive": 3703, "moodhe": 3704, "speechless": 3705, "hoarse": 3706, "approval": 3707, "romantic": 3708, "jollier": 3709, "withheld": 3710, "nobler": 3711, "skies": 3712, "chariot": 3713, "chickens": 3714, "scruff": 3715, "irate": 3716, "summoned": 3717, "ferocious": 3718, "pinned": 3719, "tool": 3720, "shed": 3721, "venture": 3722, "gardener": 3723, "remain": 3724, "stable": 3725, "gang": 3726, "slums": 3727, "pubs": 3728, "emphatic": 3729, "approbation": 3730, "oratory": 3731, "presentiment": 3732, "slice": 3733, "lemon": 3734, "debate": 3735, "assent": 3736, "assembly": 3737, "iii": 3738, "delights": 3739, "grocery": 3740, "overso": 3741, "burden": 3742, "undertook": 3743, "dining": 3744, "sixpen": 3745, "orth": 3746, "gradually": 3747, "goggles": 3748, "pa": 3749, "cord": 3750, "lift": 3751, "etc": 3752, "lifeupon": 3753, "semi": 3754, "circle": 3755, "fifth": 3756, "grovel": 3757, "gaping": 3758, "eighths": 3759, "results": 3760, "sneer": 3761, "critical": 3762, "angle": 3763, "slide": 3764, "suddenness": 3765, "thumb": 3766, "toes": 3767, "mildly": 3768, "precipitated": 3769, "flatten": 3770, "upvery": 3771, "crooked": 3772, "insecure": 3773, "smoothed": 3774, "rake": 3775, "wretchedexcept": 3776, "stepping": 3777, "corns": 3778, "surveying": 3779, "permit": 3780, "navigation": 3781, "indispensable": 3782, "track": 3783, "downright": 3784, "wisdom": 3785, "reference": 3786, "load": 3787, "swamping": 3788, "store": 3789, "essential": 3790, "pile": 3791, "servants": 3792, "host": 3793, "pence": 3794, "entertainments": 3795, "enjoys": 3796, "formalities": 3797, "fashions": 3798, "pretence": 3799, "ostentation": 3800, "withoh": 3801, "heaviest": 3802, "maddest": 3803, "dread": 3804, "neighbour": 3805, "luxuries": 3806, "bore": 3807, "criminal": 3808, "yore": 3809, "swoon": 3810, "wears": 3811, "manall": 3812, "cumbersome": 3813, "lazinessno": 3814, "skimming": 3815, "sunbeams": 3816, "flitting": 3817, "ripples": 3818, "image": 3819, "sedges": 3820, "orchis": 3821, "nots": 3822, "needa": 3823, "homely": 3824, "wear": 3825, "merchandise": 3826, "sunshinetime": 3827, "\\u00e6olian": 3828, "strings": 3829, "ustime": 3830, "fasten": 3831, "stem": 3832, "converts": 3833, "drawbacks": 3834, "expenses": 3835, "toothbrush": 3836, "powder": 3837, "determinewhen": 3838, "londonthat": 3839, "triumphed": 3840, "stumbled": 3841, "dismally": 3842, "cornered": 3843, "sharpen": 3844, "points": 3845, "huddle": 3846, "shivering": 3847, "chucks": 3848, "posture": 3849, "rock": 3850, "ugh": 3851, "kinder": 3852, "star": 3853, "crawl": 3854, "limpid": 3855, "protest": 3856, "pleasanter": 3857, "hundredweight": 3858, "flannel": 3859, "shirts": 3860, "afterbut": 3861, "shockers": 3862, "anticipate": 3863, "iv": 3864, "objections": 3865, "deserts": 3866, "provision": 3867, "retire": 3868, "significant": 3869, "ooze": 3870, "impregnating": 3871, "saturated": 3872, "spoilt": 3873, "westerly": 3874, "easterly": 3875, "northerly": 3876, "southerly": 3877, "arctic": 3878, "snows": 3879, "waste": 3880, "sands": 3881, "fragrance": 3882, "moonbeams": 3883, "positively": 3884, "reeked": 3885, "stunk": 3886, "birmingham": 3887, "steeped": 3888, "blasted": 3889, "confined": 3890, "wholesome": 3891, "quantities": 3892, "jambut": 3893, "cheesy": 3894, "buying": 3895, "ripe": 3896, "scent": 3897, "warranted": 3898, "ramshackle": 3899, "kneed": 3900, "somnambulist": 3901, "shamble": 3902, "swiftest": 3903, "roller": 3904, "whiff": 3905, "snort": 3906, "terror": 3907, "laying": 3908, "cripples": 3909, "nowhere": 3910, "porters": 3911, "marched": 3912, "proudly": 3913, "respectfully": 3914, "crusty": 3915, "rack": 3916, "squeezed": 3917, "fidget": 3918, "sniffing": 3919, "harried": 3920, "remaining": 3921, "belong": 3922, "undertaker": 3923, "depressed": 3924, "accepted": 3925, "buffet": 3926, "stamped": 3927, "responded": 3928, "compartment": 3929, "stations": 3930, "y": 3931, "mount": 3932, "stagger": 3933, "droop": 3934, "squeeze": 3935, "carriages": 3936, "euston": 3937, "understood": 3938, "detained": 3939, "moist": 3940, "madam": 3941, "ending": 3942, "residing": 3943, "detect": 3944, "melons": 3945, "injury": 3946, "guineas": 3947, "reckoning": 3948, "bargemen": 3949, "mortuary": 3950, "coroner": 3951, "plot": 3952, "deprive": 3953, "burying": 3954, "reputation": 3955, "chested": 3956, "consumptive": 3957, "throng": 3958, "declining": 3959, "sevendinner": 3960, "sticky": 3961, "concoction": 3962, "harped": 3963, "wine": 3964, "blazing": 3965, "lengthy": 3966, "parted": 3967, "victuals": 3968, "subjects": 3969, "cocked": 3970, "boss": 3971, "pushing": 3972, "teaching": 3973, "loll": 3974, "sofa": 3975, "wherever": 3976, "messing": 3977, "idle": 3978, "gaped": 3979, "slaving": 3980, "superintend": 3981, "laughedone": 3982, "crack": 3983, "jawed": 3984, "laughs": 3985, "horrible": 3986, "haunts": 3987, "perspiration": 3988, "hunt": 3989, "repack": 3990, "rummaged": 3991, "created": 3992, "chaos": 3993, "reigned": 3994, "repacked": 3995, "slammed": 3996, "pouch": 3997, "wanting": 3998, "intending": 3999, "hanged": 4000, "packer": 4001, "piles": 4002, "jars": 4003, "stoves": 4004, "strawberry": 4005, "squashed": 4006, "staring": 4007, "spinning": 4008, "teapot": 4009, "squirm": 4010, "nuisance": 4011, "stumble": 4012, "highest": 4013, "conceit": 4014, "unbearable": 4015, "laboured": 4016, "worried": 4017, "teaspoons": 4018, "lemons": 4019, "encouraged": 4020, "12": 4021, "50": 4022, "reflection": 4023, "nosix": 4024, "split": 4025, "30": 4026, "v": 4027, "arouses": 4028, "sluggard": 4029, "swindle": 4030, "depravity": 4031, "innocence": 4032, "officials": 4033, "trains": 4034, "keyhole": 4035, "oversleeping": 4036, "um": 4037, "snoring": 4038, "snarled": 4039, "interrupted": 4040, "defiant": 4041, "snore": 4042, "laythe": 4043, "uson": 4044, "maddens": 4045, "shocking": 4046, "precious": 4047, "lifethe": 4048, "againbeing": 4049, "brutish": 4050, "sloth": 4051, "inestimable": 4052, "gift": 4053, "valuable": 4054, "hereafter": 4055, "unused": 4056, "stuffing": 4057, "flirting": 4058, "slavey": 4059, "clogging": 4060, "oblivion": 4061, "resolve": 4062, "slung": 4063, "wasermarrer": 4064, "jumping": 4065, "extras": 4066, "lump": 4067, "whiling": 4068, "calmed": 4069, "fatalities": 4070, "prophesied": 4071, "storms": 4072, "midland": 4073, "counties": 4074, "tomfoolishness": 4075, "plagued": 4076, "fraud": 4077, "aggravating": 4078, "forecasts": 4079, "autumn": 4080, "report": 4081, "indoors": 4082, "wagonettes": 4083, "coaches": 4084, "shining": 4085, "chuckled": 4086, "books": 4087, "specimens": 4088, "cockle": 4089, "shells": 4090, "lark": 4091, "wetno": 4092, "cheer": 4093, "drenched": 4094, "flimsy": 4095, "rheumatism": 4096, "staying": 4097, "faster": 4098, "pointer": 4099, "prophesy": 4100, "prognosticate": 4101, "famine": 4102, "simooms": 4103, "prevented": 4104, "torrent": 4105, "overflowed": 4106, "prolonged": 4107, "spell": 4108, "printed": 4109, "oracle": 4110, "referring": 4111, "barometers": 4112, "nly": 4113, "reduce": 4114, "fahrenheit": 4115, "prophet": 4116, "horizon": 4117, "affection": 4118, "lessened": 4119, "clearing": 4120, "prophesies": 4121, "entertain": 4122, "revengeful": 4123, "portent": 4124, "proves": 4125, "especial": 4126, "readings": 4127, "atmospheric": 4128, "disturbance": 4129, "oblique": 4130, "southern": 4131, "pressure": 4132, "increasing": 4133, "cigarette": 4134, "carted": 4135, "overcoats": 4136, "macintoshes": 4137, "melon": 4138, "bulky": 4139, "talent": 4140, "securing": 4141, "unprincipled": 4142, "errand": 4143, "civilisation": 4144, "villainous": 4145, "latest": 4146, "coram": 4147, "examination": 4148, "subjected": 4149, "19": 4150, "orders": 4151, "importance": 4152, "frowned": 4153, "wounded": 4154, "touchy": 4155, "railings": 4156, "selecting": 4157, "chew": 4158, "42": 4159, "independent": 4160, "curb": 4161, "starve": 4162, "stanley": 4163, "giddy": 4164, "portion": 4165, "wedding": 4166, "bridegroom": 4167, "populace": 4168, "cabs": 4169, "belongings": 4170, "shooting": 4171, "forsake": 4172, "shying": 4173, "carrot": 4174, "traffic": 4175, "authorities": 4176, "southampton": 4177, "engine": 4178, "9": 4179, "virginia": 4180, "isle": 4181, "wight": 4182, "gents": 4183, "exeter": 4184, "mail": 4185, "stored": 4186, "tiller": 4187, "unhappy": 4188, "suspicious": 4189, "vi": 4190, "observations": 4191, "junior": 4192, "musings": 4193, "antiquity": 4194, "sheen": 4195, "maid": 4196, "wakening": 4197, "pulses": 4198, "womanhood": 4199, "villas": 4200, "grunting": 4201, "kinges": 4202, "crowned": 4203, "camped": 4204, "sloping": 4205, "uplands": 4206, "everywhere": 4207, "bess": 4208, "nuts": 4209, "virgin": 4210, "scarcely": 4211, "prime": 4212, "minister": 4213, "88": 4214, "december": 4215, "1886": 4216, "flock": 4217, "hated": 4218, "coronation": 4219, "sugar": 4220, "plums": 4221, "sack": 4222, "mead": 4223, "elgiva": 4224, "casement": 4225, "halls": 4226, "boisterous": 4227, "floated": 4228, "tumult": 4229, "odo": 4230, "dunstan": 4231, "hurl": 4232, "coarse": 4233, "insults": 4234, "brawl": 4235, "greatness": 4236, "stuarts": 4237, "moorings": 4238, "gallants": 4239, "swaggered": 4240, "ferry": 4241, "ho": 4242, "gadzooks": 4243, "gramercy": 4244, "plainly": 4245, "borough": 4246, "nobles": 4247, "courtiers": 4248, "clanking": 4249, "palfreys": 4250, "velvets": 4251, "spacious": 4252, "oriel": 4253, "fireplaces": 4254, "gabled": 4255, "roofs": 4256, "breathe": 4257, "hose": 4258, "doublet": 4259, "pearl": 4260, "embroidered": 4261, "stomachers": 4262, "upraised": 4263, "staircases": 4264, "magnificent": 4265, "mansion": 4266, "personage": 4267, "balusters": 4268, "superb": 4269, "workmanship": 4270, "decorated": 4271, "startling": 4272, "apartment": 4273, "expostulated": 4274, "match": 4275, "householder": 4276, "desiring": 4277, "enormous": 4278, "wives": 4279, "childless": 4280, "smith": 4281, "marry": 4282, "dwell": 4283, "greek": 4284, "irregular": 4285, "verbs": 4286, "unnatural": 4287, "win": 4288, "prizes": 4289, "sorts": 4290, "creature": 4291, "babe": 4292, "hay": 4293, "christmas": 4294, "stricken": 4295, "november": 4296, "toothache": 4297, "neuralgia": 4298, "ache": 4299, "chilblains": 4300, "scare": 4301, "1871": 4302, "reputed": 4303, "custards": 4304, "latin": 4305, "exercises": 4306, "grammar": 4307, "sacrificed": 4308, "fooled": 4309, "draughts": 4310, "freshened": 4311, "holidays": 4312, "whooping": 4313, "cough": 4314, "disorders": 4315, "lasted": 4316, "recommenced": 4317, "man\\u0153uvre": 4318, "oven": 4319, "artistic": 4320, "grandfathers": 4321, "commonplaces": 4322, "intrinsic": 4323, "snuffers": 4324, "prize": 4325, "halo": 4326, "glowing": 4327, "charms": 4328, "shepherds": 4329, "shepherdesses": 4330, "unvalued": 4331, "mantel": 4332, "eighteenth": 4333, "suck": 4334, "prized": 4335, "trifles": 4336, "willow": 4337, "pattern": 4338, "ranged": 4339, "chimneypieces": 4340, "2000": 4341, "rim": 4342, "species": 4343, "janes": 4344, "sheer": 4345, "mended": 4346, "bracket": 4347, "furnished": 4348, "lodgings": 4349, "amiability": 4350, "imbecility": 4351, "admiration": 4352, "200": 4353, "minus": 4354, "cabinet": 4355, "speculate": 4356, "2288": 4357, "descendants": 4358, "lovingly": 4359, "artists": 4360, "flourished": 4361, "sampler": 4362, "eldest": 4363, "spoken": 4364, "victorian": 4365, "era": 4366, "roadside": 4367, "hunted": 4368, "chipped": 4369, "claret": 4370, "travellers": 4371, "japan": 4372, "ramsgate": 4373, "souvenirs": 4374, "destruction": 4375, "jedo": 4376, "curios": 4377, "somersault": 4378, "hulloa": 4379, "admit": 4380, "violence": 4381, "coarseness": 4382, "middlesex": 4383, "separated": 4384, "lichen": 4385, "shy": 4386, "vine": 4387, "farther": 4388, "tints": 4389, "hues": 4390, "paint": 4391, "sketch": 4392, "ramble": 4393, "actual": 4394, "echo": 4395, "corridors": 4396, "creatures": 4397, "deserted": 4398, "sunlightin": 4399, "daytime": 4400, "throb": 4401, "rustle": 4402, "sighs": 4403, "bonfires": 4404, "million": 4405, "jets": 4406, "brave": 4407, "foolishhardly": 4408, "admission": 4409, "quarters": 4410, "persons": 4411, "plucked": 4412, "blessing": 4413, "losing": 4414, "largest": 4415, "bun": 4416, "whereabouts": 4417, "unanimity": 4418, "aiming": 4419, "consulted": 4420, "infuriated": 4421, "unpopular": 4422, "incapable": 4423, "reappear": 4424, "keepers": 4425, "vii": 4426, "garb": 4427, "coffins": 4428, "excepted": 4429, "busiest": 4430, "blazers": 4431, "saucy": 4432, "silken": 4433, "cloaks": 4434, "whites": 4435, "quay": 4436, "hue": 4437, "pell": 4438, "mell": 4439, "flirt": 4440, "jackets": 4441, "sparkling": 4442, "gayest": 4443, "sights": 4444, "natty": 4445, "thingsred": 4446, "matches": 4447, "necktie": 4448, "russian": 4449, "silk": 4450, "waista": 4451, "mixtures": 4452, "background": 4453, "obstinate": 4454, "oriental": 4455, "design": 4456, "frighten": 4457, "nigger": 4458, "huffy": 4459, "troubles": 4460, "prettily": 4461, "fetching": 4462, "tasteful": 4463, "excursion": 4464, "misfortune": 4465, "upall": 4466, "silky": 4467, "gloves": 4468, "photographic": 4469, "studio": 4470, "forefinger": 4471, "glove": 4472, "martyrs": 4473, "stake": 4474, "stain": 4475, "feathered": 4476, "blades": 4477, "drip": 4478, "oarsman": 4479, "flicker": 4480, "complain": 4481, "shrank": 4482, "shuddered": 4483, "unnerved": 4484, "sensitiveness": 4485, "newfoundland": 4486, "puppy": 4487, "daggers": 4488, "rollicking": 4489, "spray": 4490, "fountain": 4491, "covertly": 4492, "protect": 4493, "brushed": 4494, "tripped": 4495, "root": 4496, "agitated": 4497, "feared": 4498, "rare": 4499, "youri": 4500, "sloush": 4501, "heartedly": 4502, "tuck": 4503, "heno": 4504, "hanker": 4505, "tombstones": 4506, "recreation": 4507, "deny": 4508, "churches": 4509, "brass": 4510, "happiness": 4511, "sextons": 4512, "imperturbability": 4513, "concealed": 4514, "guarded": 4515, "smoked": 4516, "gladness": 4517, "scenethe": 4518, "elms": 4519, "hedges": 4520, "idyllic": 4521, "poetical": 4522, "inspired": 4523, "wickedness": 4524, "unconscious": 4525, "reverie": 4526, "shrill": 4527, "piping": 4528, "crying": 4529, "hobbling": 4530, "bunch": 4531, "keys": 4532, "jingled": 4533, "motioned": 4534, "screeching": 4535, "lame": 4536, "missis": 4537, "slay": 4538, "gritty": 4539, "disturb": 4540, "chock": 4541, "chivying": 4542, "yuise": 4543, "tombsgravesfolks": 4544, "knowcoffins": 4545, "untruther": 4546, "tombsnot": 4547, "kensal": 4548, "cemetery": 4549, "grandfather": 4550, "vault": 4551, "accommodating": 4552, "susan": 4553, "brick": 4554, "finchley": 4555, "headstone": 4556, "bas": 4557, "coping": 4558, "decipher": 4559, "obdurate": 4560, "fired": 4561, "hoarsely": 4562, "crypt": 4563, "fled": 4564, "sped": 4565, "revels": 4566, "monumental": 4567, "proposedsaid": 4568, "lumbering": 4569, "blowed": 4570, "cheque": 4571, "smeared": 4572, "effects": 4573, "refer": 4574, "drawer": 4575, "withdraw": 4576, "larking": 4577, "pumps": 4578, "refreshing": 4579, "beverage": 4580, "slops": 4581, "termed": 4582, "ginger": 4583, "raspberry": 4584, "syrup": 4585, "dyspepsia": 4586, "topsy": 4587, "turvy": 4588, "dived": 4589, "madder": 4590, "boorishness": 4591, "landowner": 4592, "unchristianlike": 4593, "information": 4594, "buys": 4595, "kempton": 4596, "plateau": 4597, "overhung": 4598, "coursethe": 4599, "jamwhen": 4600, "sleeves": 4601, "consideration": 4602, "definite": 4603, "hesitation": 4604, "assurance": 4605, "chummy": 4606, "abstain": 4607, "tempted": 4608, "bony": 4609, "measured": 4610, "consult": 4611, "income": 4612, "slouching": 4613, "noodles": 4614, "represent": 4615, "damage": 4616, "exertion": 4617, "firmness": 4618, "shown": 4619, "selfishness": 4620, "minor": 4621, "tributary": 4622, "rouses": 4623, "slaughter": 4624, "burn": 4625, "serve": 4626, "degenerate": 4627, "vindictiveness": 4628, "promised": 4629, "rendered": 4630, "beaming": 4631, "cheeriness": 4632, "smirking": 4633, "anticipation": 4634, "phrasing": 4635, "vocalization": 4636, "afresh": 4637, "repeating": 4638, "twell": 4639, "expectant": 4640, "meanyou": 4641, "meanthe": 4642, "arrives": 4643, "commencing": 4644, "dashes": 4645, "stops": 4646, "indeedgo": 4647, "injustice": 4648, "rankling": 4649, "considers": 4650, "favourable": 4651, "compliment": 4652, "unequal": 4653, "stronger": 4654, "nerved": 4655, "dawnedlaughing": 4656, "coursei": 4657, "jenkins": 4658, "cellar": 4659, "warnings": 4660, "office": 4661, "attorney": 4662, "falsetto": 4663, "audience": 4664, "floorno": 4665, "iti": 4666, "pardonfunny": 4667, "iand": 4668, "ioh": 4669, "de": 4670, "chorusit": 4671, "dee": 4672, "harm": 4673, "imagines": 4674, "parties": 4675, "incident": 4676, "fashionable": 4677, "cultured": 4678, "happyall": 4679, "germany": 4680, "uncomfortable": 4681, "tastes": 4682, "morceaux": 4683, "masters": 4684, "philosophy": 4685, "ethics": 4686, "flirted": 4687, "humorousin": 4688, "recited": 4689, "sentimental": 4690, "ballad": 4691, "spanish": 4692, "weepit": 4693, "reciting": 4694, "funnier": 4695, "funnythat": 4696, "seriousness": 4697, "irresistibly": 4698, "amuse": 4699, "unobtrusive": 4700, "accompanied": 4701, "soulful": 4702, "guess": 4703, "ignorance": 4704, "artful": 4705, "continuously": 4706, "scowled": 4707, "fiercely": 4708, "convulsions": 4709, "mock": 4710, "seriousnessoh": 4711, "surpassed": 4712, "glowered": 4713, "ferocity": 4714, "forewarned": 4715, "wailing": 4716, "wept": 4717, "amid": 4718, "germans": 4719, "translate": 4720, "danced": 4721, "fists": 4722, "hartz": 4723, "lover": 4724, "jilted": 4725, "spiriti": 4726, "details": 4727, "acknowledged": 4728, "tragic": 4729, "situation": 4730, "usvery": 4731, "unostentatious": 4732, "avoiding": 4733, "sunbury": 4734, "crosses": 4735, "splendidly": 4736, "rhythmical": 4737, "tiniest": 4738, "peep": 4739, "considerate": 4740, "sully": 4741, "natured": 4742, "waltona": 4743, "entrenchment": 4744, "bradshaw": 4745, "sojourned": 4746, "scold": 4747, "bridle": 4748, "curbing": 4749, "tongues": 4750, "scarce": 4751, "tremendously": 4752, "fee": 4753, "duchess": 4754, "york": 4755, "graveyard": 4756, "epitaph": 4757, "inscribed": 4758, "thereon": 4759, "deserve": 4760, "corway": 4761, "bridgewas": 4762, "planting": 4763, "halliford": 4764, "adroit": 4765, "movement": 4766, "clumsiness": 4767, "weybridge": 4768, "wey": 4769, "navigable": 4770, "guildford": 4771, "explore": 4772, "bourne": 4773, "basingstoke": 4774, "enter": 4775, "closer": 4776, "barking": 4777, "fallen": 4778, "oilskin": 4779, "parcel": 4780, "instruction": 4781, "ix": 4782, "introduced": 4783, "heathenish": 4784, "speed": 4785, "callous": 4786, "prone": 4787, "consciencenot": 4788, "conscienceobject": 4789, "patience": 4790, "fold": 4791, "revolting": 4792, "knots": 4793, "loops": 4794, "honourable": 4795, "exceptions": 4796, "professionconscientious": 4797, "linestow": 4798, "crochet": 4799, "knit": 4800, "antimacassars": 4801, "sincerely": 4802, "looped": 4803, "lifted": 4804, "scientifically": 4805, "unravel": 4806, "swaddling": 4807, "infant": 4808, "unwound": 4809, "mat": 4810, "net": 4811, "grunts": 4812, "muddle": 4813, "unwind": 4814, "exclaims": 4815, "scaffolding": 4816, "hauling": 4817, "tighter": 4818, "climbs": 4819, "countenance": 4820, "act": 4821, "truant": 4822, "incidents": 4823, "briskly": 4824, "animated": 4825, "vainly": 4826, "shrieking": 4827, "distress": 4828, "politely": 4829, "tomdick": 4830, "affably": 4831, "dunder": 4832, "springs": 4833, "roars": 4834, "pitch": 4835, "remembering": 4836, "frequent": 4837, "chattering": 4838, "offering": 4839, "resistance": 4840, "reminding": 4841, "oblivious": 4842, "discussing": 4843, "adrift": 4844, "absorbing": 4845, "overtaken": 4846, "hulking": 4847, "chaps": 4848, "restraining": 4849, "clasped": 4850, "auntie": 4851, "tower": 4852, "shelves": 4853, "reposeful": 4854, "ripping": 4855, "linen": 4856, "sheets": 4857, "larboard": 4858, "reclined": 4859, "disembarked": 4860, "starboard": 4861, "hooks": 4862, "carpet": 4863, "urging": 4864, "gallop": 4865, "realised": 4866, "mishap": 4867, "fashionand": 4868, "docould": 4869, "similar": 4870, "misfortunes": 4871, "risk": 4872, "theirs": 4873, "hitched": 4874, "overturns": 4875, "cuts": 4876, "giggles": 4877, "undo": 4878, "twist": 4879, "strangled": 4880, "drifts": 4881, "pinning": 4882, "occurs": 4883, "pin": 4884, "stopgo": 4885, "ongo": 4886, "asks": 4887, "knowdon": 4888, "wayyou": 4889, "chivy": 4890, "important": 4891, "shutting": 4892, "runnymead": 4893, "afterward": 4894, "figurative": 4895, "ladycousin": 4896, "sideand": 4897, "inat": 4898, "dusk": 4899, "lockwallingfordand": 4900, "hesitatingly": 4901, "excessive": 4902, "punishment": 4903, "reassure": 4904, "marked": 4905, "reliable": 4906, "gathering": 4907, "hobgoblins": 4908, "banshees": 4909, "wisps": 4910, "pools": 4911, "hymns": 4912, "reflections": 4913, "thenfar": 4914, "orpheus": 4915, "lute": 4916, "apollo": 4917, "harrowed": 4918, "correctly": 4919, "spasmodically": 4920, "variations": 4921, "accordion": 4922, "reassuring": 4923, "alongside": 4924, "provincial": 4925, "arrys": 4926, "arriets": 4927, "attractive": 4928, "sounding": 4929, "faust": 4930, "x": 4931, "contrariness": 4932, "virtuous": 4933, "pacific": 4934, "dragging": 4935, "tons": 4936, "originally": 4937, "inlets": 4938, "earlier": 4939, "calledand": 4940, "elm": 4941, "spreading": 4942, "roots": 4943, "dispensed": 4944, "bargained": 4945, "abstract": 4946, "croquet": 4947, "estimate": 4948, "kick": 4949, "nipped": 4950, "wrestling": 4951, "hoop": 4952, "covering": 4953, "bungled": 4954, "superhuman": 4955, "tucked": 4956, "freedomthe": 4957, "birthright": 4958, "englishman": 4959, "considerably": 4960, "interfere": 4961, "troublesome": 4962, "involved": 4963, "wriggling": 4964, "mummy": 4965, "suffocated": 4966, "withstand": 4967, "undid": 4968, "cleared": 4969, "decks": 4970, "sputtering": 4971, "loudly": 4972, "overhear": 4973, "insteadtea": 4974, "boils": 4975, "trickery": 4976, "breadth": 4977, "clank": 4978, "cutlery": 4979, "crockery": 4980, "sets": 4981, "molars": 4982, "bumped": 4983, "fullhow": 4984, "conscience": 4985, "digested": 4986, "mealso": 4987, "domination": 4988, "digestive": 4989, "organs": 4990, "unless": 4991, "wills": 4992, "dictates": 4993, "emotions": 4994, "passions": 4995, "spoonsful": 4996, "brain": 4997, "quivering": 4998, "soar": 4999, "whirling": 5000, "lanes": 5001, "flaming": 5002, "eternity": 5003, "muffins": 5004, "beast": 5005, "fielda": 5006, "brainless": 5007, "listless": 5008, "unlit": 5009, "laughdrivel": 5010, "folly": 5011, "ninny": 5012, "kittens": 5013, "alcohol": 5014, "sorriest": 5015, "slaves": 5016, "morality": 5017, "righteousness": 5018, "vigilantly": 5019, "unsought": 5020, "citizen": 5021, "fathera": 5022, "snappy": 5023, "corn": 5024, "wishes": 5025, "shudder": 5026, "ware": 5027, "wheat": 5028, "observing": 5029, "treading": 5030, "advising": 5031, "thisaway": 5032, "temptation": 5033, "possibility": 5034, "drains": 5035, "wales": 5036, "slightly": 5037, "lurched": 5038, "grope": 5039, "itone": 5040, "crawling": 5041, "compass": 5042, "valiantly": 5043, "doleful": 5044, "pig": 5045, "bangs": 5046, "novelty": 5047, "hardness": 5048, "cramped": 5049, "nightfor": 5050, "morningkept": 5051, "digging": 5052, "spine": 5053, "owe": 5054, "otherwise": 5055, "excruciating": 5056, "wrench": 5057, "ached": 5058, "aboutsome": 5059, "sisterconversing": 5060, "mysteries": 5061, "childish": 5062, "worship": 5063, "echoing": 5064, "dome": 5065, "spans": 5066, "vista": 5067, "hoping": 5068, "hovering": 5069, "sorrows": 5070, "fevered": 5071, "smiles": 5072, "flushed": 5073, "moan": 5074, "mightier": 5075, "goodly": 5076, "briars": 5077, "sorely": 5078, "grieving": 5079, "mourning": 5080, "journeying": 5081, "logs": 5082, "comrade": 5083, "ragged": 5084, "beggar": 5085, "shone": 5086, "radiance": 5087, "befallen": 5088, "torn": 5089, "nigh": 5090, "devious": 5091, "entranced": 5092, "faded": 5093, "kneeling": 5094, "saint": 5095, "forest": 5096, "xi": 5097, "heroism": 5098, "moral": 5099, "historical": 5100, "retrospect": 5101, "inserted": 5102, "schools": 5103, "watches": 5104, "absurdity": 5105, "gippings": 5106, "occurrence": 5107, "shortest": 5108, "ministers": 5109, "defend": 5110, "shaved": 5111, "anathematized": 5112, "decent": 5113, "unlocked": 5114, "holborn": 5115, "shutter": 5116, "bus": 5117, "cart": 5118, "dilapidated": 5119, "stooped": 5120, "eyeing": 5121, "neighbouring": 5122, "constable": 5123, "guardian": 5124, "severely": 5125, "redressing": 5126, "wakeful": 5127, "enliven": 5128, "horribly": 5129, "undisguised": 5130, "lanterns": 5131, "slinking": 5132, "doorways": 5133, "regulation": 5134, "flip": 5135, "flop": 5136, "distrustful": 5137, "rout": 5138, "constables": 5139, "key": 5140, "scuttleful": 5141, "coals": 5142, "dropping": 5143, "burglars": 5144, "detectives": 5145, "handcuff": 5146, "morbidly": 5147, "pictured": 5148, "believing": 5149, "sentenced": 5150, "penal": 5151, "servitude": 5152, "overcoat": 5153, "finishing": 5154, "prod": 5155, "aid": 5156, "sending": 5157, "poked": 5158, "shivered": 5159, "overnight": 5160, "fling": 5161, "shawls": 5162, "joyous": 5163, "delicious": 5164, "tempting": 5165, "precedence": 5166, "vent": 5167, "horrors": 5168, "sorted": 5169, "snags": 5170, "weeds": 5171, "wormed": 5172, "dipped": 5173, "surface": 5174, "duffers": 5175, "accidentally": 5176, "awfully": 5177, "drivelling": 5178, "imbecile": 5179, "ar": 5180, "n": 5181, "youyougoing": 5182, "peals": 5183, "shirtit": 5184, "springing": 5185, "deuce": 5186, "picnics": 5187, "yachts": 5188, "pined": 5189, "eggsor": 5190, "harassing": 5191, "flicking": 5192, "fingers": 5193, "performing": 5194, "feat": 5195, "culinary": 5196, "indian": 5197, "sandwich": 5198, "incantations": 5199, "scalded": 5200, "anticipated": 5201, "teaspoonful": 5202, "unappetizing": 5203, "aids": 5204, "housekeeping": 5205, "remind": 5206, "1215": 5207, "sons": 5208, "homespun": 5209, "dirk": 5210, "witness": 5211, "writing": 5212, "stupendous": 5213, "page": 5214, "translated": 5215, "oliver": 5216, "morningsunny": 5217, "thrill": 5218, "echoed": 5219, "clang": 5220, "horses": 5221, "captains": 5222, "surly": 5223, "jests": 5224, "bearded": 5225, "bowmen": 5226, "billmen": 5227, "pikemen": 5228, "foreign": 5229, "spearmen": 5230, "companies": 5231, "ridden": 5232, "travel": 5233, "townsmen": 5234, "woe": 5235, "betide": 5236, "plaintiff": 5237, "executioner": 5238, "tempestuous": 5239, "pays": 5240, "sparing": 5241, "pleases": 5242, "bellow": 5243, "roystering": 5244, "gamble": 5245, "quarrel": 5246, "deepens": 5247, "firelight": 5248, "sheds": 5249, "uncouth": 5250, "brawny": 5251, "wenches": 5252, "bandy": 5253, "jest": 5254, "jibe": 5255, "swaggering": 5256, "troopers": 5257, "unlike": 5258, "swains": 5259, "despised": 5260, "apart": 5261, "grins": 5262, "broad": 5263, "glitter": 5264, "camps": 5265, "followers": 5266, "mustered": 5267, "hover": 5268, "wolves": 5269, "sentinel": 5270, "twinkling": 5271, "fires": 5272, "ages": 5273, "workmen": 5274, "pavilion": 5275, "yester": 5276, "eve": 5277, "carpenters": 5278, "nailing": 5279, "tiers": 5280, "prentices": 5281, "stuffs": 5282, "guttural": 5283, "stalwart": 5284, "halbert": 5285, "menbarons": 5286, "theseand": 5287, "halt": 5288, "bands": 5289, "casques": 5290, "breastplates": 5291, "steeds": 5292, "horsemen": 5293, "galloping": 5294, "banners": 5295, "fluttering": 5296, "baron": 5297, "serfs": 5298, "vassals": 5299, "slope": 5300, "cooper": 5301, "rustics": 5302, "townsfolk": 5303, "version": 5304, "event": 5305, "coracleswhich": 5306, "poorer": 5307, "rapids": 5308, "sturdy": 5309, "crowding": 5310, "fateful": 5311, "waits": 5312, "noon": 5313, "stolen": 5314, "heels": 5315, "charters": 5316, "wriggled": 5317, "risen": 5318, "pattering": 5319, "hoofs": 5320, "pushes": 5321, "cavalcade": 5322, "lords": 5323, "flank": 5324, "rides": 5325, "greets": 5326, "honeyed": 5327, "dismount": 5328, "casts": 5329, "hem": 5330, "unsuspecting": 5331, "horseman": 5332, "unready": 5333, "rebellious": 5334, "rue": 5335, "thwart": 5336, "bolder": 5337, "richard": 5338, "sinks": 5339, "rein": 5340, "dismounts": 5341, "foremost": 5342, "mailed": 5343, "hilt": 5344, "swift": 5345, "ponderous": 5346, "grumble": 5347, "grate": 5348, "cleaves": 5349, "cornerstone": 5350, "xii": 5351, "disadvantages": 5352, "nation": 5353, "search": 5354, "homeless": 5355, "houseless": 5356, "prepares": 5357, "conjuring": 5358, "remarked": 5359, "recalled": 5360, "prosaic": 5361, "tuft": 5362, "polishing": 5363, "commit": 5364, "customer": 5365, "ankerwyke": 5366, "spooning": 5367, "thrall": 5368, "photographs": 5369, "relatives": 5370, "pausing": 5371, "coldly": 5372, "papa": 5373, "items": 5374, "news": 5375, "opinions": 5376, "closes": 5377, "shuts": 5378, "relied": 5379, "civilised": 5380, "community": 5381, "poke": 5382, "buckinghamshire": 5383, "unexpectedly": 5384, "mooning": 5385, "wraysbury": 5386, "blushed": 5387, "billing": 5388, "cooing": 5389, "albansnice": 5390, "kissing": 5391, "marriage": 5392, "ouseley": 5393, "drunkso": 5394, "confessor": 5395, "proved": 5396, "encompassed": 5397, "choked": 5398, "nearing": 5399, "albert": 5400, "victoria": 5401, "diggings": 5402, "clematis": 5403, "creeper": 5404, "honey": 5405, "suckle": 5406, "itno": 5407, "theredidn": 5408, "thereharris": 5409, "informant": 5410, "ideals": 5411, "hollowness": 5412, "traps": 5413, "traveller": 5414, "occasion": 5415, "needn": 5416, "sensibly": 5417, "greeting": 5418, "fourteenth": 5419, "stables": 5420, "cellars": 5421, "roughing": 5422, "itshe": 5423, "mindbut": 5424, "eton": 5425, "panting": 5426, "bargeman": 5427, "enlivened": 5428, "pains": 5429, "assumed": 5430, "producing": 5431, "unattractive": 5432, "messenger": 5433, "paralysed": 5434, "pigstye": 5435, "disused": 5436, "limekiln": 5437, "placeat": 5438, "powered": 5439, "sustain": 5440, "fainted": 5441, "seize": 5442, "consciousness": 5443, "roomed": 5444, "mothergood": 5445, "allfive": 5446, "poundsand": 5447, "pots": 5448, "2ft": 5449, "6in": 5450, "truckle": 5451, "bathed": 5452, "tugged": 5453, "tackled": 5454, "seldom": 5455, "universe": 5456, "spoonful": 5457, "reckless": 5458, "extravagant": 5459, "offers": 5460, "absurdly": 5461, "value": 5462, "mountain": 5463, "switzerland": 5464, "shanty": 5465, "francs": 5466, "scandalous": 5467, "childhood": 5468, "opener": 5469, "spiky": 5470, "uninjured": 5471, "teacup": 5472, "poised": 5473, "dangers": 5474, "brings": 5475, "anew": 5476, "exaggerations": 5477, "hammered": 5478, "flattened": 5479, "geometrybut": 5480, "unearthly": 5481, "hideousness": 5482, "dent": 5483, "snobby": 5484, "haunt": 5485, "overdressed": 5486, "showy": 5487, "chiefly": 5488, "dudes": 5489, "witch": 5490, "riversteam": 5491, "journal": 5492, "duke": 5493, "volume": 5494, "dines": 5495, "spree": 5496, "leisurely": 5497, "clieveden": 5498, "blended": 5499, "unbroken": 5500, "lingeringly": 5501, "stiffish": 5502, "sprung": 5503, "upin": 5504, "veers": 5505, "consistently": 5506, "ways": 5507, "probation": 5508, "sparks": 5509, "upward": 5510, "bellied": 5511, "steered": 5512, "thrilling": 5513, "yetexcept": 5514, "onward": 5515, "plodding": 5516, "puny": 5517, "tortuously": 5518, "throbbing": 5519, "raising": 5520, "limbs": 5521, "brothers": 5522, "skimmed": 5523, "intently": 5524, "tinged": 5525, "towering": 5526, "enchantment": 5527, "ecstatic": 5528, "purple": 5529, "gloaming": 5530, "legend": 5531, "beings": 5532, "sorting": 5533, "usnot": 5534, "comprehensive": 5535, "embraced": 5536, "career": 5537, "included": 5538, "connected": 5539, "usgood": 5540, "boatsbetter": 5541, "xiii": 5542, "decides": 5543, "civil": 5544, "receipts": 5545, "hindering": 5546, "pleasantest": 5547, "centres": 5548, "bustling": 5549, "neverthelessstanding": 5550, "travels": 5551, "owned": 5552, "algar": 5553, "conquering": 5554, "matilda": 5555, "earls": 5556, "paget": 5557, "councillor": 5558, "successive": 5559, "sovereigns": 5560, "climbing": 5561, "glades": 5562, "scented": 5563, "vistas": 5564, "fairer": 5565, "rung": 5566, "cleves": 5567, "melodramatic": 5568, "properties": 5569, "holy": 5570, "walks": 5571, "trivial": 5572, "kingdoms": 5573, "salisbury": 5574, "poitiers": 5575, "inspecting": 5576, "monuments": 5577, "beeches": 5578, "shelley": 5579, "west": 5580, "composed": 5581, "revolt": 5582, "islam": 5583, "quote": 5584, "phraseology": 5585, "sebert": 5586, "offa": 5587, "invading": 5588, "encamped": 5589, "gloucestershire": 5590, "hell": 5591, "commonly": 5592, "notorious": 5593, "wilkes": 5594, "motto": 5595, "invitation": 5596, "doorway": 5597, "bogus": 5598, "congregation": 5599, "irreverent": 5600, "jesters": 5601, "monastery": 5602, "sterner": 5603, "revellers": 5604, "cistercian": 5605, "thirteenth": 5606, "tunics": 5607, "cowls": 5608, "mass": 5609, "themthe": 5610, "whisperings": 5611, "windshould": 5612, "truer": 5613, "myriad": 5614, "greenlands": 5615, "residence": 5616, "newsagenta": 5617, "unassuming": 5618, "regions": 5619, "genially": 5620, "throughuntil": 5621, "tolerably": 5622, "pussy": 5623, "tickle": 5624, "sticks": 5625, "rigid": 5626, "wipes": 5627, "gentleness": 5628, "meets": 5629, "contenting": 5630, "clouting": 5631, "christians": 5632, "reformation": 5633, "rowdiness": 5634, "shopping": 5635, "mastiff": 5636, "collies": 5637, "bernard": 5638, "retrievers": 5639, "newfoundlands": 5640, "hound": 5641, "mangy": 5642, "lowther": 5643, "arcade": 5644, "animals": 5645, "tykes": 5646, "peacefulness": 5647, "calmness": 5648, "resignationof": 5649, "pervaded": 5650, "chained": 5651, "dreamlessly": 5652, "haughty": 5653, "shadow": 5654, "provocation": 5655, "fore": 5656, "vigorously": 5657, "attacked": 5658, "foxey": 5659, "impartial": 5660, "willing": 5661, "canine": 5662, "hearths": 5663, "homes": 5664, "depended": 5665, "fray": 5666, "indiscriminately": 5667, "biting": 5668, "pandemonium": 5669, "terrific": 5670, "vestry": 5671, "murdered": 5672, "poles": 5673, "riot": 5674, "lamb": 5675, "nasty": 5676, "brutes": 5677, "darted": 5678, "joythe": 5679, "warrior": 5680, "handsthe": 5681, "uttered": 5682, "scots": 5683, "hilland": 5684, "sinewy": 5685, "updid": 5686, "trotted": 5687, "inquiring": 5688, "chilled": 5689, "boldest": 5690, "abruptly": 5691, "nonot": 5692, "allcertainlydon": 5693, "allquite": 5694, "thanksnot": 5695, "allvery": 5696, "fitting": 5697, "groove": 5698, "unimportant": 5699, "shrink": 5700, "piteously": 5701, "marketing": 5702, "revictualled": 5703, "vegetablesthat": 5704, "bushel": 5705, "gooseberry": 5706, "tarts": 5707, "mutton": 5708, "foraged": 5709, "successes": 5710, "impressive": 5711, "ostentatious": 5712, "spectacle": 5713, "curs": 5714, "bulged": 5715, "lime": 5716, "baker": 5717, "confectioner": 5718, "haired": 5719, "cheesemonger": 5720, "fruiterer": 5721, "stray": 5722, "informing": 5723, "numbers": 5724, "houseboats": 5725, "strangle": 5726, "blatant": 5727, "bumptiousness": 5728, "knack": 5729, "rousing": 5730, "hatchet": 5731, "arrows": 5732, "breach": 5733, "lordly": 5734, "ensure": 5735, "verdict": 5736, "justifiable": 5737, "homicide": 5738, "boastful": 5739, "delay": 5740, "aggravation": 5741, "sighting": 5742, "anecdote": 5743, "reverse": 5744, "engines": 5745, "narrative": 5746, "instruct": 5747, "rightyou": 5748, "oneleave": 5749, "younow": 5750, "aristocratic": 5751, "beanfeast": 5752, "messrs": 5753, "cubit": 5754, "bermondsey": 5755, "saucepan": 5756, "windsora": 5757, "mechanical": 5758, "monstrositieswith": 5759, "description": 5760, "glimpse": 5761, "families": 5762, "hardy": 5763, "spokesman": 5764, "wherewhere": 5765, "brand": 5766, "pump": 5767, "chancing": 5768, "germs": 5769, "poison": 5770, "boiling": 5771, "settling": 5772, "westward": 5773, "quietest": 5774, "peacefullest": 5775, "contentedmore": 5776, "bodied": 5777, "developed": 5778, "serene": 5779, "abreast": 5780, "cosily": 5781, "marsh": 5782, "saving": 5783, "studded": 5784, "surrounded": 5785, "menacing": 5786, "torture": 5787, "imprisonment": 5788, "dares": 5789, "watersi": 5790, "boors": 5791, "claim": 5792, "threaten": 5793, "itbut": 5794, "feed": 5795, "hundreds": 5796, "tumbled": 5797, "descending": 5798, "practicable": 5799, "froze": 5800, "veins": 5801, "headand": 5802, "headsticking": 5803, "deadand": 5804, "heredarn": 5805, "rescuing": 5806, "pievery": 5807, "harristumbled": 5808, "grubby": 5809, "gully": 5810, "conjecture": 5811, "believes": 5812, "planned": 5813, "unjust": 5814, "calumny": 5815, "xiv": 5816, "waxworks": 5817, "studies": 5818, "difficulties": 5819, "troubled": 5820, "mellowed": 5821, "bends": 5822, "lingers": 5823, "retina": 5824, "dragon": 5825, "boasts": 5826, "r": 5827, "ilk": 5828, "depicted": 5829, "author": 5830, "andmore": 5831, "stillwas": 5832, "bequeathed": 5833, "annually": 5834, "divided": 5835, "easter": 5836, "undutiful": 5837, "untruths": 5838, "rumoured": 5839, "thingsor": 5840, "themand": 5841, "wax": 5842, "tennyson": 5843, "placid": 5844, "rustic": 5845, "arry": 5846, "fitznoodle": 5847, "vanished": 5848, "mortar": 5849, "roses": 5850, "bursting": 5851, "splendour": 5852, "veritable": 5853, "courtyard": 5854, "politics": 5855, "roamed": 5856, "undertaking": 5857, "cheerfully": 5858, "skittishly": 5859, "peeled": 5860, "leftat": 5861, "itit": 5862, "pea": 5863, "nut": 5864, "scraped": 5865, "potatoesall": 5866, "warts": 5867, "hollows": 5868, "scrapings": 5869, "economy": 5870, "cabbage": 5871, "peck": 5872, "remnants": 5873, "pork": 5874, "potted": 5875, "salmon": 5876, "advantage": 5877, "thicken": 5878, "ingredients": 5879, "evinced": 5880, "strolled": 5881, "reappearing": 5882, "contribution": 5883, "genuine": 5884, "experiments": 5885, "progress": 5886, "piquant": 5887, "palate": 5888, "nourishing": 5889, "softer": 5890, "poema": 5891, "cherry": 5892, "manifested": 5893, "challenge": 5894, "threatening": 5895, "spit": 5896, "growled": 5897, "nosed": 5898, "scoundrel": 5899, "spout": 5900, "mixture": 5901, "growl": 5902, "rapid": 5903, "goodsaid": 5904, "soothed": 5905, "nerves": 5906, "twanged": 5907, "evenings": 5908, "unnerve": 5909, "retort": 5910, "postpone": 5911, "sorryfor": 5912, "himbut": 5913, "practising": 5914, "captured": 5915, "evidence": 5916, "bound": 5917, "elapsed": 5918, "coldnessthe": 5919, "despaired": 5920, "advertised": 5921, "sacrifice": 5922, "card": 5923, "disheartening": 5924, "studying": 5925, "contend": 5926, "unfeelingly": 5927, "committed": 5928, "mercy": 5929, "gurgle": 5930, "successful": 5931, "precautions": 5932, "affect": 5933, "shark": 5934, "guineawhere": 5935, "earshot": 5936, "confessed": 5937, "perform": 5938, "startat": 5939, "magnificently": 5940, "collapsed": 5941, "hiss": 5942, "complaints": 5943, "insufficiency": 5944, "repertoirenone": 5945, "campbells": 5946, "hoorayhooray": 5947, "scotland": 5948, "guesses": 5949, "disagreeable": 5950, "regatta": 5951, "homeas": 5952, "coldish": 5953, "conjured": 5954, "shapeless": 5955, "giant": 5956, "glow": 5957, "snug": 5958, "pecking": 5959, "chunks": 5960, "knives": 5961, "filling": 5962, "overflowing": 5963, "prior": 5964, "uncertainties": 5965, "striking": 5966, "response": 5967, "hopefully": 5968, "hallooed": 5969, "crammed": 5970, "knocking": 5971, "cottagers": 5972, "householders": 5973, "apartments": 5974, "assaulting": 5975, "hits": 5976, "refuses": 5977, "despairingly": 5978, "skin": 5979, "babes": 5980, "hopeyes": 5981, "novels": 5982, "resolved": 5983, "strictly": 5984, "truthful": 5985, "employ": 5986, "phrases": 5987, "glimmer": 5988, "flickering": 5989, "thenoh": 5990, "divinest": 5991, "sleepersi": 5992, "sleepers": 5993, "oneand": 5994, "blackness": 5995, "tiredness": 5996, "screaming": 5997, "safely": 5998, "nest": 5999, "defended": 6000, "paddled": 6001, "sleepily": 6002, "facts": 6003, "trials": 6004, "examined": 6005, "wandering": 6006, "hazy": 6007, "remembrance": 6008, "hearing": 6009, "muttering": 6010, "xv": 6011, "duties": 6012, "tells": 6013, "scepticism": 6014, "recollections": 6015, "beginner": 6016, "friendship": 6017, "housework": 6018, "non": 6019, "dainties": 6020, "continual": 6021, "afford": 6022, "insight": 6023, "posed": 6024, "menamely": 6025, "manages": 6026, "chime": 6027, "fascinates": 6028, "breaks": 6029, "passion": 6030, "wing": 6031, "possession": 6032, "preservation": 6033, "crave": 6034, "scrupulous": 6035, "due": 6036, "crew": 6037, "ridiculed": 6038, "hegeorge": 6039, "himselfwho": 6040, "skulks": 6041, "hadmost": 6042, "fully": 6043, "compelled": 6044, "support": 6045, "rejoined": 6046, "passenger": 6047, "superintended": 6048, "slaved": 6049, "arranging": 6050, "similarly": 6051, "cushions": 6052, "encourages": 6053, "marvellous": 6054, "drawls": 6055, "whiffs": 6056, "perspiring": 6057, "biffles": 6058, "afternoonnever": 6059, "partially": 6060, "wakes": 6061, "recollects": 6062, "remembers": 6063, "unusually": 6064, "waylikewise": 6065, "speaker": 6066, "reprovingly": 6067, "exhausted": 6068, "conversational": 6069, "oarsmen": 6070, "elders": 6071, "digest": 6072, "faith": 6073, "wegeorge": 6074, "myselftook": 6075, "un": 6076, "plied": 6077, "customary": 6078, "onesthe": 6079, "honoured": 6080, "pastand": 6081, "invented": 6082, "episode": 6083, "degree": 6084, "oursa": 6085, "mocked": 6086, "recounting": 6087, "oarsmanship": 6088, "contributing": 6089, "threepence": 6090, "regent": 6091, "drying": 6092, "lodge": 6093, "acquired": 6094, "suburban": 6095, "brickfieldsan": 6096, "providing": 6097, "materials": 6098, "equal": 6099, "intimately": 6100, "acquainted": 6101, "superfluous": 6102, "reluctant": 6103, "accepting": 6104, "proof": 6105, "coolness": 6106, "dodges": 6107, "flattering": 6108, "advances": 6109, "youthful": 6110, "legged": 6111, "inevitable": 6112, "interview": 6113, "mostly": 6114, "exclamatory": 6115, "mono": 6116, "syllabic": 6117, "proficient": 6118, "clubs": 6119, "afternoons": 6120, "handling": 6121, "swamped": 6122, "acquiring": 6123, "prompt": 6124, "admired": 6125, "hiring": 6126, "richmond": 6127, "named": 6128, "serpentine": 6129, "tide": 6130, "select": 6131, "oared": 6132, "racing": 6133, "ardour": 6134, "launched": 6135, "shoved": 6136, "disappear": 6137, "magic": 6138, "broadside": 6139, "dipping": 6140, "entertainment": 6141, "arch": 6142, "renewed": 6143, "sobs": 6144, "prefers": 6145, "eastbourne": 6146, "flourishing": 6147, "parade": 6148, "nobility": 6149, "gentry": 6150, "secured": 6151, "fretful": 6152, "vehement": 6153, "striving": 6154, "bane": 6155, "straining": 6156, "overtakes": 6157, "annoy": 6158, "overtake": 6159, "himall": 6160, "sublime": 6161, "equanimity": 6162, "ordeal": 6163, "uppishness": 6164, "youngster": 6165, "twentieth": 6166, "disentangles": 6167, "explains": 6168, "adapt": 6169, "limited": 6170, "capacity": 6171, "devote": 6172, "setting": 6173, "moderate": 6174, "inspiration": 6175, "answers": 6176, "willingly": 6177, "assisting": 6178, "exchange": 6179, "notnot": 6180, "recovery": 6181, "mutual": 6182, "abuse": 6183, "friendly": 6184, "sympathetic": 6185, "cheeky": 6186, "plant": 6187, "punter": 6188, "unfortunately": 6189, "undignified": 6190, "lagging": 6191, "precaution": 6192, "scramble": 6193, "streampossibly": 6194, "lent": 6195, "novice": 6196, "spun": 6197, "bets": 6198, "outcome": 6199, "exhibition": 6200, "bounds": 6201, "chaff": 6202, "unmercifully": 6203, "reprove": 6204, "ridiculing": 6205, "ribaldry": 6206, "deriding": 6207, "jeering": 6208, "peppered": 6209, "stale": 6210, "private": 6211, "perfectly": 6212, "unintelligible": 6213, "jibes": 6214, "decency": 6215, "deem": 6216, "excused": 6217, "boulogne": 6218, "forcibly": 6219, "whoever": 6220, "hercules": 6221, "unavailing": 6222, "captor": 6223, "regained": 6224, "emerged": 6225, "stammered": 6226, "confusedly": 6227, "relation": 6228, "outright": 6229, "toothough": 6230, "rounders": 6231, "sport": 6232, "yare": 6233, "luff": 6234, "luffed": 6235, "hurricane": 6236, "hectori": 6237, "namewent": 6238, "funerals": 6239, "boom": 6240, "ships": 6241, "sopping": 6242, "vexing": 6243, "downmore": 6244, "sideways": 6245, "likeand": 6246, "painter": 6247, "arriving": 6248, "phenomenon": 6249, "obstinacy": 6250, "suicide": 6251, "disappoint": 6252, "exhausting": 6253, "seafaring": 6254, "lashed": 6255, "main": 6256, "jib": 6257, "squalls": 6258, "easiest": 6259, "contrived": 6260, "embrace": 6261, "travelled": 6262, "sailed": 6263, "heeled": 6264, "righted": 6265, "miracle": 6266, "ploughed": 6267, "according": 6268, "bladder": 6269, "surfeit": 6270, "saila": 6271, "sailand": 6272, "rescued": 6273, "ignominious": 6274, "tipping": 6275, "xvi": 6276, "shirk": 6277, "anchored": 6278, "warships": 6279, "kennet": 6280, "ravage": 6281, "praying": 6282, "westminster": 6283, "courts": 6284, "lawyers": 6285, "besieged": 6286, "essex": 6287, "prince": 6288, "james": 6289, "benedictine": 6290, "gaunt": 6291, "blanche": 6292, "confoundedly": 6293, "impertinent": 6294, "tilehurst": 6295, "hardwick": 6296, "bowls": 6297, "habitues": 6298, "exhibitions": 6299, "loose": 6300, "unreasonable": 6301, "neared": 6302, "blanched": 6303, "prematurely": 6304, "stamp": 6305, "pinch": 6306, "poverty": 6307, "uswe": 6308, "coroners": 6309, "courtssome": 6310, "deceivedor": 6311, "sinnedsome": 6312, "thenand": 6313, "closed": 6314, "millstone": 6315, "drudgery": 6316, "procured": 6317, "remainder": 6318, "unitedly": 6319, "bond": 6320, "plainer": 6321, "spectre": 6322, "respectability": 6323, "erring": 6324, "outcast": 6325, "childhad": 6326, "betraying": 6327, "chocolate": 6328, "bitterest": 6329, "centred": 6330, "hug": 6331, "stabs": 6332, "gall": 6333, "shadowed": 6334, "deeps": 6335, "dusky": 6336, "robe": 6337, "sinned": 6338, "thingssinned": 6339, "sinners": 6340, "woo": 6341, "barrier": 6342, "affirm": 6343, "villages": 6344, "choice": 6345, "xvii": 6346, "fisher": 6347, "superintendence": 6348, "wearable": 6349, "themwell": 6350, "cleaner": 6351, "dirt": 6352, "washerwoman": 6353, "excavating": 6354, "abounds": 6355, "roach": 6356, "eels": 6357, "minnows": 6358, "district": 6359, "shoals": 6360, "itnot": 6361, "thrower": 6362, "gumption": 6363, "imagination": 6364, "shocker": 6365, "reporter": 6366, "invention": 6367, "possess": 6368, "ability": 6369, "fabrication": 6370, "tyro": 6371, "circumstantial": 6372, "embellishing": 6373, "probability": 6374, "scrupulousalmost": 6375, "pedanticveracity": 6376, "experienced": 6377, "weighing": 6378, "measuring": 6379, "appropriates": 6380, "puff": 6381, "lets": 6382, "brag": 6383, "momentary": 6384, "lull": 6385, "removes": 6386, "knocks": 6387, "calmly": 6388, "tinge": 6389, "bitterness": 6390, "refills": 6391, "continues": 6392, "shouldn": 6393, "literally": 6394, "nothingexcept": 6395, "hourhalf": 6396, "snap": 6397, "surprisedi": 6398, "astonishment": 6399, "buggles": 6400, "missus": 6401, "listens": 6402, "hauls": 6403, "add": 6404, "threeat": 6405, "increased": 6406, "percentage": 6407, "simplify": 6408, "moderation": 6409, "disadvantage": 6410, "anglers": 6411, "jealous": 6412, "fishyou": 6413, "foundation": 6414, "lately": 6415, "committee": 6416, "association": 6417, "adoption": 6418, "older": 6419, "sipping": 6420, "indigestion": 6421, "shave": 6422, "pipeclaying": 6423, "sincegeorge": 6424, "ensued": 6425, "chimney": 6426, "fascinated": 6427, "monstrous": 6428, "cod": 6429, "uncommon": 6430, "ounces": 6431, "minnow": 6432, "remarkably": 6433, "genial": 6434, "lockleastways": 6435, "thenone": 6436, "whopper": 6437, "aback": 6438, "bleak": 6439, "individual": 6440, "comer": 6441, "forgive": 6442, "weperfect": 6443, "neighbourhoodare": 6444, "thingmost": 6445, "guessing": 6446, "scale": 6447, "histories": 6448, "immensely": 6449, "bates": 6450, "muggles": 6451, "jones": 6452, "billy": 6453, "maunders": 6454, "honest": 6455, "plays": 6456, "wag": 6457, "bringing": 6458, "whacking": 6459, "astonishing": 6460, "marvelled": 6461, "alarm": 6462, "fragmentsi": 6463, "paris": 6464, "xviii": 6465, "photographed": 6466, "drowning": 6467, "demoralizing": 6468, "extraordinarily": 6469, "cleve": 6470, "longest": 6471, "teddington": 6472, "eights": 6473, "regretted": 6474, "seeker": 6475, "depths": 6476, "sinking": 6477, "strip": 6478, "widens": 6479, "prison": 6480, "welcoming": 6481, "eyed": 6482, "287": 6483, "exchanged": 6484, "fairyland": 6485, "speculative": 6486, "hurriedly": 6487, "ruffle": 6488, "rakish": 6489, "assuming": 6490, "affability": 6491, "fan": 6492, "frowning": 6493, "agility": 6494, "forehead": 6495, "wistfulness": 6496, "cynicism": 6497, "eventful": 6498, "rightat": 6499, "altered": 6500, "squinted": 6501, "noses": 6502, "pushed": 6503, "stentorian": 6504, "woodwork": 6505, "tilting": 6506, "ordained": 6507, "madly": 6508, "undoubtedly": 6509, "foreground": 6510, "bits": 6511, "insignificant": 6512, "paltry": 6513, "compared": 6514, "subscribe": 6515, "bespoke": 6516, "rescinded": 6517, "negative": 6518, "unpleasantness": 6519, "tenths": 6520, "britons": 6521, "evicted": 6522, "replaced": 6523, "trace": 6524, "sweeping": 6525, "masons": 6526, "halted": 6527, "crumbled": 6528, "saxons": 6529, "normans": 6530, "walled": 6531, "fortified": 6532, "siege": 6533, "fairfax": 6534, "razed": 6535, "hilly": 6536, "varied": 6537, "paddling": 6538, "delightfully": 6539, "drowsiness": 6540, "caer": 6541, "doren": 6542, "recent": 6543, "capital": 6544, "nods": 6545, "hampden": 6546, "wonderfully": 6547, "exception": 6548, "quaintest": 6549, "gables": 6550, "timeyfied": 6551, "divinely": 6552, "bump": 6553, "unexpected": 6554, "operation": 6555, "impossibility": 6556, "surprising": 6557, "featherbed": 6558, "monotonous": 6559, "culhalm": 6560, "lockthe": 6561, "coldest": 6562, "deepest": 6563, "riverthe": 6564, "improves": 6565, "typical": 6566, "smaller": 6567, "orderquiet": 6568, "eminently": 6569, "desperately": 6570, "prides": 6571, "compare": 6572, "doubtful": 6573, "sanctified": 6574, "brew": 6575, "nowadays": 6576, "nicholas": 6577, "monument": 6578, "blackwall": 6579, "jane": 6580, "helen": 6581, "1637": 6582, "issue": 6583, "loins": 6584, "lacking": 6585, "numbered": 6586, "ninety": 6587, "leefive": 6588, "mayor": 6589, "abingdonwas": 6590, "benefactor": 6591, "overcrowded": 6592, "courteney": 6593, "visit": 6594, "viewed": 6595, "tuesdays": 6596, "thursdays": 6597, "curiosities": 6598, "pool": 6599, "lasher": 6600, "undercurrent": 6601, "marks": 6602, "diving": 6603, "favourite": 6604, "brethren": 6605, "disappointing": 6606, "fairish": 6607, "drives": 6608, "college": 6609, "exceptionally": 6610, "irritable": 6611, "mishaps": 6612, "occur": 6613, "indulgently": 6614, "behave": 6615, "mildest": 6616, "gentlest": 6617, "imaginable": 6618, "unfortunate": 6619, "sculler": 6620, "brutally": 6621, "amiable": 6622, "demoralising": 6623, "calmer": 6624, "regret": 6625, "xix": 6626, "beauties": 6627, "changes": 6628, "yearnings": 6629, "flight": 6630, "whichever": 6631, "squaring": 6632, "contemplate": 6633, "boatunless": 6634, "handled": 6635, "rarely": 6636, "sink": 6637, "arrangementsor": 6638, "allto": 6639, "ornamental": 6640, "airs": 6641, "chiefone": 6642, "recommendation": 6643, "modest": 6644, "likes": 6645, "hides": 6646, "boata": 6647, "names": 6648, "struggling": 6649, "antediluvian": 6650, "recently": 6651, "carelessly": 6652, "relics": 6653, "surmise": 6654, "geologist": 6655, "pooh": 6656, "poohed": 6657, "meanest": 6658, "category": 6659, "include": 6660, "evidences": 6661, "proving": 6662, "preglacial": 6663, "pre": 6664, "adamite": 6665, "humorous": 6666, "reward": 6667, "persisted": 6668, "sharply": 6669, "tub": 6670, "builder": 6671, "boatwas": 6672, "selected": 6673, "whitewashed": 6674, "tarredhad": 6675, "distinguish": 6676, "offended": 6677, "stock": 6678, "pasted": 6679, "shabbier": 6680, "prayers": 6681, "loan": 6682, "remnant": 6683, "homeward": 6684, "drizzle": 6685, "riverwith": 6686, "wavelets": 6687, "gilding": 6688, "beech": 6689, "chasing": 6690, "flinging": 6691, "diamonds": 6692, "wheels": 6693, "kisses": 6694, "wantoning": 6695, "silvering": 6696, "bridges": 6697, "townlet": 6698, "inlet": 6699, "gleaming": 6700, "gloryis": 6701, "riverchill": 6702, "ceaseless": 6703, "weeping": 6704, "shrouded": 6705, "mists": 6706, "vapour": 6707, "reproachful": 6708, "actions": 6709, "neglectedis": 6710, "regrets": 6711, "melancholy": 6712, "enthusiastic": 6713, "storm": 6714, "soberly": 6715, "hoisted": 6716, "poured": 6717, "persistency": 6718, "clammy": 6719, "veal": 6720, "apt": 6721, "whitebait": 6722, "cutlet": 6723, "babbled": 6724, "soles": 6725, "sauce": 6726, "nap": 6727, "fourpencegeorge": 6728, "cardsand": 6729, "gambling": 6730, "breeds": 6731, "revenge": 6732, "saddest": 6733, "volunteers": 6734, "aldershot": 6735, "cripple": 6736, "introduce": 6737, "sciatica": 6738, "fevers": 6739, "chills": 6740, "lung": 6741, "frolicksome": 6742, "vein": 6743, "extracted": 6744, "weep": 6745, "yearnful": 6746, "unutterable": 6747, "jaw": 6748, "abandon": 6749, "rendering": 6750, "bedthat": 6751, "undressed": 6752, "slumber": 6753, "pour": 6754, "mackintoshes": 6755, "usi": 6756, "myselfmade": 6757, "attempts": 6758, "expressing": 6759, "sentiments": 6760, "unnecessary": 6761, "climate": 6762, "disastrous": 6763, "prospect": 6764, "finish": 6765, "almanac": 6766, "skirt": 6767, "venturing": 6768, "survey": 6769, "311": 6770, "unconsciously": 6771, "contract": 6772, "deaths": 6773, "casting": 6774, "malevolence": 6775, "mention": 6776, "figures": 6777, "shamed": 6778, "stealthily": 6779, "gaudy": 6780, "mackintosh": 6781, "instructions": 6782, "unforeseen": 6783, "paddington": 6784, "leicester": 6785, "presenting": 6786, "paybox": 6787, "informed": 6788, "renowned": 6789, "contortionists": 6790, "himalaya": 6791, "greater": 6792, "bronzed": 6793, "countenances": 6794, "cynosure": 6795, "awaiting": 6796, "burgundy": 6797, "sauces": 6798, "loaves": 6799, "welcome": 6800, "pegged": 6801, "quaffed": 6802, "carelesslywhen": 6803, "critically": 6804, "smoky": 6805, "hitherto": 6806, "dowhen": 6807, "curtain": 6808, "glistened": 6809, "darkly": 6810, "lamps": 6811, "flickered": 6812, "gust": 6813, "puddles": 6814, "trickled": 6815, "spouts": 6816, "gutters": 6817, "wayfarers": 6818, "dripping": 6819, "skirts": 6820, "thamesbut": 6821, "hind": 6822, "concurrence": 6823}'

In [14]:

# Save word_index and index_word as python dictionaries

index_word = json.loads(tokenizer_config['index_word'])
word_index = json.loads(tokenizer_config['word_index'])

Map the sentences to tokens¶

You can map each sentence to a sequence of integer tokens using the Tokenizer's texts_to_sequences() method. As was the case for the IMDb data set, the number corresponding to a word is that word's frequency rank in the corpus.

In [15]:

# View the first 5 sentences
sentence_strings[:5]

Out[15]:

['CHAPTER I',
 '   Three invalids',
 'Sufferings of George and Harris',
 'A victim to one hundred and seven fatal maladies',
 'Useful prescriptions']

In [16]:

# Tokenize the data
sentence_seq = tokenizer.texts_to_sequences(sentence_strings)

In [17]:

# The return type is a list
type(sentence_seq)

Out[17]:

list

In [18]:

# View the first 5 tokenized sentences
sentence_seq[0:5]

Out[18]:

[[362, 8],
 [126, 3362],
 [2319, 6, 36, 3, 35],
 [5, 1779, 4, 43, 363, 3, 468, 3363, 2320],
 [2321, 3364]]

In [19]:

# Verify the mappings in the config
print(word_index['chapter'], word_index['i'])
print(word_index['three'], word_index['invalids'])
print (
    word_index['sufferings'], word_index['of'], word_index['george'],
    word_index['and'], word_index['harris']
)
print (
    word_index['a'], word_index['victim'], word_index['to'], word_index['one'], word_index['hundred'],
    word_index['and'], word_index['seven'], word_index['fatal'], word_index['maladies']
)
print(word_index['useful'], word_index['prescriptions'])

362 8
126 3362
2319 6 36 3 35
5 1779 4 43 363 3 468 3363 2320
2321 3364

Map the tokens to sentences¶

You can map the tokens back to sentences using the Tokenizer's sequences_to_texts method.

In [20]:

# View the first 5 tokenized sentences
sentence_seq[0:5]

Out[20]:

[[362, 8],
 [126, 3362],
 [2319, 6, 36, 3, 35],
 [5, 1779, 4, 43, 363, 3, 468, 3363, 2320],
 [2321, 3364]]

In [21]:

# Map the token sequences back to sentences
tokenizer.sequences_to_texts(sentence_seq)[:5]

Out[21]:

['chapter i',
 'three invalids',
 'sufferings of george and harris',
 'a victim to one hundred and seven fatal maladies',
 'useful prescriptions']

In [22]:

# Verify the mappings in the config
print(index_word['362'], index_word['8'])
print(index_word['126'], index_word['3362'])
print(index_word['2319'], index_word['6'], index_word['36'], index_word['3'], index_word['35'])
print (
    index_word['5'], index_word['1779'], index_word['4'], index_word['43'], index_word['363'],
    index_word['3'], index_word['468'], index_word['3363'], index_word['2320']
)
print(index_word['2321'], index_word['3364'])

chapter i
three invalids
sufferings of george and harris
a victim to one hundred and seven fatal maladies
useful prescriptions

In [23]:

# Any valid sequence of tokens can be converted to text
tokenizer.sequences_to_texts([[92, 104, 241], [152, 169, 53, 2491]])

Out[23]:

['good day world', 'montmorency bit my finger']

If a word is not featured in the Tokenizer's word index, then it will be mapped to the value of the Tokenizer's oov_token property.

In [24]:

# Tokenize unrecognised words
tokenizer.texts_to_sequences(['i would like goobleydoobly hobbledyho'])

Out[24]:

[[8, 28, 78, 1, 1]]

In [25]:

# Verify the OOV token
index_word['1']

Out[25]:

'<UNK>'

Further reading and resources¶

• https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer
• https://nlp.stanford.edu/IR-book/html/htmledition/tokenization-1.html

Stateful RNNs

In this reading notebook you will learn how to retain the state of an RNN when processing long sequences.

In [1]:

import tensorflow as tf
tf.__version__

Out[1]:

'2.2.0'

So far you have trained RNNs on entire sequences, possibly of varying length. In some applications, such as financial time series modeling or real-time speech processing, the input sequence can be very long.

One way to process such sequences is to simply chop up the sequences into separate batches. However, the internal state of the RNN would then normally be reset in between the batches. Persisting an RNN cell's state between batches is useful in such contexts.

Stateful and non-stateful RNN models¶

We will begin by creating two versions of the same RNN model. The first is a regular RNN that does not retain its state.

In [2]:

# Create a regular (non-stateful) RNN
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import GRU
gru = Sequential([GRU(5, input_shape = (None, 1), name = 'rnn')])

To persist RNN cell states between batches, you can use the stateful argument when you initialize an RNN layer. The default value of this argument is False. This argument is available for all RNN layer types.

In [3]:

# Create a stateful RNN
stateful_gru = Sequential([GRU(5, stateful = True, batch_input_shape = (2, None, 1), name = 'stateful_rnn')])

Note that as well as setting stateful=True, we have also specified the batch_input_shape. This fixes the number of elements in a batch, as well as providing the sequence length and number of features. So the above model will always require a batch of 2 sequences.

When using stateful RNNs, it is necessary to supply this argument to the first layer of a Sequential model. This is because the model will always assume that each element of every subsequent batch it receives will be a continuation of the sequence from the corresponding element in the previous batch.

Another detail is that when defining a model with a stateful RNN using the functional API, you will need to specify the batch_shape argument as follows:

In [4]:

# Redefine the same stateful RNN using the functional API
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input
inputs = Input(batch_shape = (2, None, 1))
outputs = GRU(5, stateful = True, name = 'stateful_rnn')(inputs)
stateful_gru = Model(inputs = inputs, outputs = outputs)

Inspect the RNN states¶

We can inspect the RNN layer states by retrieving the recurrent layer from each model, and looking at the states property.

In [5]:

# Retrieve the RNN layer and inspect the internal state
gru.get_layer('rnn').states

Out[5]:

[None]

In [6]:

# Retrieve the RNN layer and inspect the internal state
stateful_gru.get_layer('stateful_rnn').states

Out[6]:

[<tf.Variable 'stateful_rnn_1/Variable:0' shape=(2, 5) dtype=float32, numpy=
 array([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]], dtype=float32)>]

Note that the internal state of the stateful RNN has a state stored for each element in a batch, which is why the shape of the state Variable is (2, 5).

Create a simple sequence dataset¶

We will demonstrate the effect of statefulness on a simple sequence dataset consisting of two sequences.

In [7]:

# Create the sequence dataset
sequence_data = tf.constant (
    [
        [[-4.], [-3.], [-2.], [-1.], [0.], [1.], [2.], [3.], [4.]],
        [[-40.], [-30.], [-20.], [-10.], [0.], [10.], [20.], [30.], [40.]]
    ],
    dtype = tf.float32
)
sequence_data.shape

Out[7]:

TensorShape([2, 9, 1])

Process the sequence batch with both models¶

Now see what happens when you pass the batch of sequences through either model:

In [8]:

# Process the batch with both models
_1 = gru(sequence_data)
_2 = stateful_gru(sequence_data)

In [9]:

# Retrieve the RNN layer and inspect the internal state
gru.get_layer('rnn').states

Out[9]:

[None]

In [10]:

# Retrieve the RNN layer and inspect the internal state
stateful_gru.get_layer('stateful_rnn').states

Out[10]:

[<tf.Variable 'stateful_rnn_1/Variable:0' shape=(2, 5) dtype=float32, numpy=
 array([[ 0.62783915,  0.25128558, -0.8052597 , -0.86494243, -0.3519494 ],
        [-0.47144485, -0.4271023 , -0.99999565, -1.        ,  0.32106256]],
       dtype=float32)>]

The stateful RNN model has updated and retained its state after having processed the input sequence batch. This internal state could then be used as the initial state for processing a continuation of both sequences in the next batch.

Resetting the internal state¶

If you need a stateful RNN to forget (or re-initialise) its state, then you can call an RNN layer's reset_states() method.

In [11]:

# Reset the internal state of the stateful RNN model
stateful_gru.get_layer('stateful_rnn').reset_states()

In [12]:

# Retrieve the RNN layer and inspect the internal state
stateful_gru.get_layer('stateful_rnn').states

Out[12]:

[<tf.Variable 'stateful_rnn_1/Variable:0' shape=(2, 5) dtype=float32, numpy=
 array([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]], dtype=float32)>]

Note that reset_states() resets the state to 0., which is the default initial state for the RNN layers in TensorFlow.

Retaining internal state across batches¶

Passing a sequence to a stateful layer as several subsequences produces the same final output as passing the whole sequence at once.

In [13]:

# Reset the internal state of the stateful RNN model and process the full sequences
stateful_gru.get_layer('stateful_rnn').reset_states()
_ = stateful_gru(sequence_data)
stateful_gru.get_layer('stateful_rnn').states

Out[13]:

[<tf.Variable 'stateful_rnn_1/Variable:0' shape=(2, 5) dtype=float32, numpy=
 array([[ 0.62783915,  0.25128558, -0.8052597 , -0.86494243, -0.3519494 ],
        [-0.47144485, -0.4271023 , -0.99999565, -1.        ,  0.32106256]],
       dtype=float32)>]

In [14]:

# Break the sequences into batches

sequence_batch1 = sequence_data[:, :3, :]
sequence_batch2 = sequence_data[:, 3:6, :]
sequence_batch3 = sequence_data[:, 6:, :]

print("First batch:", sequence_batch1)
print("\nSecond batch:", sequence_batch2)
print("\nThird batch:", sequence_batch3)

First batch: tf.Tensor(
[[[ -4.]
  [ -3.]
  [ -2.]]

 [[-40.]
  [-30.]
  [-20.]]], shape=(2, 3, 1), dtype=float32)

Second batch: tf.Tensor(
[[[ -1.]
  [  0.]
  [  1.]]

 [[-10.]
  [  0.]
  [ 10.]]], shape=(2, 3, 1), dtype=float32)

Third batch: tf.Tensor(
[[[ 2.]
  [ 3.]
  [ 4.]]

 [[20.]
  [30.]
  [40.]]], shape=(2, 3, 1), dtype=float32)

Note that the first element in every batch is part of the same sequence, and the second element in every batch is part of the same sequence.

In [15]:

# Reset the internal state of the stateful RNN model and process the batches in order
stateful_gru.get_layer('stateful_rnn').reset_states()
_ = stateful_gru(sequence_batch1)
_ = stateful_gru(sequence_batch2)
_ = stateful_gru(sequence_batch3)
stateful_gru.get_layer('stateful_rnn').states

Out[15]:

[<tf.Variable 'stateful_rnn_1/Variable:0' shape=(2, 5) dtype=float32, numpy=
 array([[ 0.62783915,  0.25128558, -0.8052597 , -0.86494243, -0.3519494 ],
        [-0.47144485, -0.4271023 , -0.99999565, -1.        ,  0.32106256]],
       dtype=float32)>]

Notice that the internal state of the stateful RNN after processing each batch is the same as it was earlier when we processed the entire sequence at once.

This property can be used when training stateful RNNs, if we ensure that each example in a batch is a continuation of the same sequence as the corresponding example in the previous batch.

Further reading and resources¶

• https://www.tensorflow.org/guide/keras/rnn#cross-batch_statefulness

Sequence Modelling

In [1]:

import tensorflow as tf
print(tf.__version__)

2.2.0

1. The IMDb dataset¶

2. Padding and masking sequence data¶

3. The Embedding layer¶

4. The Embedding Projector¶

5. Recurrent neural network layers¶

6. Stacked RNNs and the Bidirectional wrapper¶

---

The IMDb Dataset¶

Load the IMDB review sentiment dataset¶

In [2]:

# Import imdb
import tensorflow.keras.datasets.imdb as imdb

In [3]:

# Download and assign the data set using load_data()
(x_train, y_train), (x_test, y_test) = imdb.load_data()

/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/imdb.py:155: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/imdb.py:156: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])

Inspect the dataset¶

In [4]:

# Inspect the type of the data
type(x_train)

Out[4]:

numpy.ndarray

In [5]:

# Inspect the shape of the data
x_train.shape

Out[5]:

(25000,)

In [6]:

# Display the first dataset element input
# Notice encoding
print(x_train[0])

[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 22665, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 21631, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 19193, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 10311, 8, 4, 107, 117, 5952, 15, 256, 4, 31050, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 12118, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]

In [7]:

# Display the first dataset element output
y_train[0]

Out[7]:

1

Load dataset with different options¶

In [ ]:

# Load the dataset with defaults
imdb.load_data(path = 'imdb.npz', index_from = 3)
# ~/.keras/dataset/

In [ ]:

# Limit the vocabulary to the top 500 words using num_words
imdb.load_data(num_words = 1000)

In [ ]:

# Ignore the top 10 most frequent words using skip_top
imdb.load_data(skip_top = 10, num_words = 1000, oov_char = 2)

In [ ]:

# Limit the sequence lengths to 500 using maxlen
imdb.load_data(maxlen = 500)

In [ ]:

# Use '1' as the character that indicates the start of a sequence
imdb.load_data(start_char = 1)

Explore the dataset word index¶

In [8]:

# Load the imdb word index using get_word_index()
imdb_word_index = imdb.get_word_index()

In [9]:

# View the word index as a dictionary,
# accounting for index_from.
index_from = 3
imdb_word_index = {key: value + index_from for key, value in imdb_word_index.items()}

In [10]:

# Retrieve a specific word's index
imdb_word_index['simpsonian']

Out[10]:

52256

In [11]:

# View an input sentence
inv_imdb_word_index = {value: key for key, value in imdb_word_index.items()}
print([inv_imdb_word_index[index] for index in x_train[0] if index > index_from])

['this', 'film', 'was', 'just', 'brilliant', 'casting', 'location', 'scenery', 'story', 'direction', "everyone's", 'really', 'suited', 'the', 'part', 'they', 'played', 'and', 'you', 'could', 'just', 'imagine', 'being', 'there', 'robert', "redford's", 'is', 'an', 'amazing', 'actor', 'and', 'now', 'the', 'same', 'being', 'director', "norman's", 'father', 'came', 'from', 'the', 'same', 'scottish', 'island', 'as', 'myself', 'so', 'i', 'loved', 'the', 'fact', 'there', 'was', 'a', 'real', 'connection', 'with', 'this', 'film', 'the', 'witty', 'remarks', 'throughout', 'the', 'film', 'were', 'great', 'it', 'was', 'just', 'brilliant', 'so', 'much', 'that', 'i', 'bought', 'the', 'film', 'as', 'soon', 'as', 'it', 'was', 'released', 'for', 'retail', 'and', 'would', 'recommend', 'it', 'to', 'everyone', 'to', 'watch', 'and', 'the', 'fly', 'fishing', 'was', 'amazing', 'really', 'cried', 'at', 'the', 'end', 'it', 'was', 'so', 'sad', 'and', 'you', 'know', 'what', 'they', 'say', 'if', 'you', 'cry', 'at', 'a', 'film', 'it', 'must', 'have', 'been', 'good', 'and', 'this', 'definitely', 'was', 'also', 'congratulations', 'to', 'the', 'two', 'little', "boy's", 'that', 'played', 'the', "part's", 'of', 'norman', 'and', 'paul', 'they', 'were', 'just', 'brilliant', 'children', 'are', 'often', 'left', 'out', 'of', 'the', 'praising', 'list', 'i', 'think', 'because', 'the', 'stars', 'that', 'play', 'them', 'all', 'grown', 'up', 'are', 'such', 'a', 'big', 'profile', 'for', 'the', 'whole', 'film', 'but', 'these', 'children', 'are', 'amazing', 'and', 'should', 'be', 'praised', 'for', 'what', 'they', 'have', 'done', "don't", 'you', 'think', 'the', 'whole', 'story', 'was', 'so', 'lovely', 'because', 'it', 'was', 'true', 'and', 'was', "someone's", 'life', 'after', 'all', 'that', 'was', 'shared', 'with', 'us', 'all']

In [12]:

# Get the sentiment value
y_train[0]

Out[12]:

1

---

Padding and Masking Sequence Data¶

In [13]:

# Load the imdb data set
import tensorflow.keras.datasets.imdb as imdb
(x_train, y_train), (x_test, y_test) = imdb.load_data()

Preprocess the data with padding¶

In [14]:

# Inspect the input data shape
x_train.shape

Out[14]:

(25000,)

In [15]:

# Pad the inputs to the maximum length using maxlen
padded_x_train = tf.keras.preprocessing.sequence.pad_sequences (
    x_train, maxlen = 300, padding = 'post', truncating = 'pre'
)

In [16]:

# Inspect the output data shape
padded_x_train.shape

Out[16]:

(25000, 300)

Create a Masking layer¶

In [17]:

# Import numpy 
import numpy as np

In [18]:

# Masking expects to see (batch, sequence, features)
# Create a dummy feature dimension using expand_dims
padded_x_train = np.expand_dims(padded_x_train, -1)

In [19]:

# Create a Masking layer 
tf_x_train = tf.convert_to_tensor(padded_x_train, dtype = 'float32')
masking_layer = tf.keras.layers.Masking(mask_value = 0.0)

In [20]:

# Pass tf_x_train to it
masked_x_train = masking_layer(tf_x_train)

In [21]:

# Look at the dataset
masked_x_train

Out[21]:

<tf.Tensor: shape=(25000, 300, 1), dtype=float32, numpy=
array([[[1.000e+00],
        [1.400e+01],
        [2.200e+01],
        ...,
        [0.000e+00],
        [0.000e+00],
        [0.000e+00]],

       [[1.000e+00],
        [1.940e+02],
        [1.153e+03],
        ...,
        [0.000e+00],
        [0.000e+00],
        [0.000e+00]],

       [[1.000e+00],
        [1.400e+01],
        [4.700e+01],
        ...,
        [0.000e+00],
        [0.000e+00],
        [0.000e+00]],

       ...,

       [[1.000e+00],
        [1.100e+01],
        [6.000e+00],
        ...,
        [0.000e+00],
        [0.000e+00],
        [0.000e+00]],

       [[1.000e+00],
        [1.446e+03],
        [7.079e+03],
        ...,
        [0.000e+00],
        [0.000e+00],
        [0.000e+00]],

       [[1.000e+00],
        [1.700e+01],
        [6.000e+00],
        ...,
        [0.000e+00],
        [0.000e+00],
        [0.000e+00]]], dtype=float32)>

In [22]:

# Look at the ._keras_mask for the dataset
masked_x_train._keras_mask

Out[22]:

<tf.Tensor: shape=(25000, 300), dtype=bool, numpy=
array([[ True,  True,  True, ..., False, False, False],
       [ True,  True,  True, ..., False, False, False],
       [ True,  True,  True, ..., False, False, False],
       ...,
       [ True,  True,  True, ..., False, False, False],
       [ True,  True,  True, ..., False, False, False],
       [ True,  True,  True, ..., False, False, False]])>

---

The Embedding layer¶

Create and apply an Embedding layer¶

In [23]:

# Create an embedding layer using layers.Embedding
# Specify input_dim, output_dim, input_length
embedding_layer = tf.keras.layers.Embedding(input_dim = 501, output_dim = 16)

In [24]:

# Inspect an Embedding layer output for a fixed input
# Expects an input of shape (batch, sequence, feature)
sequence_of_indices = tf.constant([[[0], [1], [5], [500]]])
sequence_of_embeddings = embedding_layer(sequence_of_indices)
sequence_of_embeddings

Out[24]:

<tf.Tensor: shape=(1, 4, 1, 16), dtype=float32, numpy=
array([[[[-0.04408611,  0.04542727,  0.02113073, -0.00829327,
          -0.01037695, -0.02869653,  0.04612013, -0.0363334 ,
          -0.04592133,  0.03994573, -0.0116153 ,  0.0039704 ,
           0.04329038, -0.01370787,  0.00340534, -0.0268536 ]],

        [[-0.01728276,  0.02123139,  0.03937277, -0.00543173,
          -0.03721107,  0.02353149,  0.0325414 ,  0.01882902,
           0.00865491,  0.03644737, -0.00558869,  0.03472593,
          -0.03631423, -0.01145393, -0.03586975, -0.04967504]],

        [[ 0.00114288,  0.0259436 ,  0.03903201, -0.00509673,
          -0.01917675,  0.01715554, -0.02361081,  0.04791243,
           0.03532455,  0.01695074,  0.02925735, -0.0171978 ,
           0.02004195, -0.00264467,  0.03264482,  0.0466736 ]],

        [[-0.00167334, -0.01974974,  0.03862996, -0.00783445,
          -0.0119449 ,  0.02677094, -0.04435721, -0.02603542,
           0.01661992, -0.01721267,  0.01067184,  0.00779436,
          -0.00333594,  0.03251171, -0.00892366,  0.02862989]]]],
      dtype=float32)>

In [25]:

# Inspect the Embedding layer weights using get_weights()
embedding_layer.get_weights()[0]

Out[25]:

array([[-0.04408611,  0.04542727,  0.02113073, ..., -0.01370787,
         0.00340534, -0.0268536 ],
       [-0.01728276,  0.02123139,  0.03937277, ..., -0.01145393,
        -0.03586975, -0.04967504],
       [ 0.03113865,  0.03436014, -0.03613175, ..., -0.02740414,
        -0.03701732,  0.03566268],
       ...,
       [-0.01433843, -0.03155677,  0.0031219 , ...,  0.01358536,
        -0.02125506,  0.04881435],
       [-0.03752189, -0.01023849,  0.00854632, ...,  0.00272838,
         0.04810517, -0.02201123],
       [-0.00167334, -0.01974974,  0.03862996, ...,  0.03251171,
        -0.00892366,  0.02862989]], dtype=float32)

In [26]:

# Get the embedding for the 14th index
embedding_layer.get_weights()[0][14, :]

Out[26]:

array([ 0.02291333, -0.02097397,  0.01462842,  0.00676034,  0.01067505,
        0.00024272,  0.03100765, -0.01003016, -0.0367582 , -0.04832541,
       -0.03168309,  0.01910916, -0.00131863, -0.01287366, -0.02852354,
        0.02369959], dtype=float32)

Create and apply an Embedding layer that uses mask_zero=True¶

In [27]:

# Create a layer that uses the mask_zero kwarg
masking_embedding_layer = tf.keras.layers.Embedding(input_dim = 501, output_dim = 16, mask_zero = True)

In [28]:

# Apply this layer to the sequence and see the _keras_mask property
masked_sequence_of_embeddings = masking_embedding_layer(sequence_of_indices)
masked_sequence_of_embeddings._keras_mask

Out[28]:

<tf.Tensor: shape=(1, 4, 1), dtype=bool, numpy=
array([[[False],
        [ True],
        [ True],
        [ True]]])>

---

The Embedding Projector¶

Load and preprocess the IMDb data¶

In [29]:

# A function to load and preprocess the IMDB dataset
def get_and_pad_imdb_dataset(num_words = 10000, maxlen = None, index_from = 2):
    from tensorflow.keras.datasets import imdb

    # Load the reviews
    (x_train, y_train), (x_test, y_test) = imdb.load_data (
        path = 'imdb.npz', num_words = num_words, skip_top = 0,
        maxlen = maxlen, start_char = 1, oov_char = 2, index_from = index_from
    )

    x_train = tf.keras.preprocessing.sequence.pad_sequences (
        x_train, maxlen = None, padding = 'pre', truncating = 'pre', value = 0
    )

    x_test = tf.keras.preprocessing.sequence.pad_sequences (
        x_test, maxlen = None, padding = 'pre', truncating = 'pre', value = 0
    )

    return (x_train, y_train), (x_test, y_test)

In [30]:

# Load the dataset
(x_train, y_train), (x_test, y_test) = get_and_pad_imdb_dataset()

In [31]:

# A function to get the dataset word index
def get_imdb_word_index(num_words = 10000, index_from = 2):
    imdb_word_index = tf.keras.datasets.imdb.get_word_index(path = 'imdb_word_index.json')
    imdb_word_index = {
        key: value + index_from for key, value in imdb_word_index.items() if value <= num_words-index_from
    }
    return imdb_word_index

In [32]:

# Get the word index
imdb_word_index = get_imdb_word_index()

In [33]:

# Swap the keys and values of the word index
inv_imdb_word_index = {value: key for key, value in imdb_word_index.items()}

In [34]:

# View the first dataset example sentence
print([inv_imdb_word_index[index] for index in x_train[100] if index > 2])

['i', 'am', 'a', 'great', 'fan', 'of', 'david', 'lynch', 'and', 'have', 'everything', 'that', "he's", 'made', 'on', 'dvd', 'except', 'for', 'hotel', 'room', 'the', '2', 'hour', 'twin', 'peaks', 'movie', 'so', 'when', 'i', 'found', 'out', 'about', 'this', 'i', 'immediately', 'grabbed', 'it', 'and', 'and', 'what', 'is', 'this', "it's", 'a', 'bunch', 'of', 'drawn', 'black', 'and', 'white', 'cartoons', 'that', 'are', 'loud', 'and', 'foul', 'mouthed', 'and', 'unfunny', 'maybe', 'i', "don't", 'know', "what's", 'good', 'but', 'maybe', 'this', 'is', 'just', 'a', 'bunch', 'of', 'crap', 'that', 'was', 'on', 'the', 'public', 'under', 'the', 'name', 'of', 'david', 'lynch', 'to', 'make', 'a', 'few', 'bucks', 'too', 'let', 'me', 'make', 'it', 'clear', 'that', 'i', "didn't", 'care', 'about', 'the', 'foul', 'language', 'part', 'but', 'had', 'to', 'keep', 'the', 'sound', 'because', 'my', 'neighbors', 'might', 'have', 'all', 'in', 'all', 'this', 'is', 'a', 'highly', 'disappointing', 'release', 'and', 'may', 'well', 'have', 'just', 'been', 'left', 'in', 'the', 'box', 'set', 'as', 'a', 'curiosity', 'i', 'highly', 'recommend', 'you', "don't", 'spend', 'your', 'money', 'on', 'this', '2', 'out', 'of', '10']

Build an Embedding layer into a model¶

In [35]:

# Get the maximum token value
max_index_value = max(imdb_word_index.values())

In [36]:

# Specify an embedding dimension
embedding_dim = 16

In [37]:

# Build a model using Sequential:
#     1. Embedding layer
#     2. GlobalAveragePooling1D
#     3. Dense
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(input_dim = max_index_value + 1, output_dim = embedding_dim, mask_zero = False),
    tf.keras.layers.GlobalAveragePooling1D(),
    tf.keras.layers.Dense(units = 1, activation = 'sigmoid'),
])

In [38]:

# Functional API refresher: use the Model to build the same model
review_sequence = tf.keras.Input((None,))
embedding_sequence = tf.keras.layers.Embedding(input_dim = max_index_value + 1, output_dim = embedding_dim)(review_sequence)
average_sequence = tf.keras.layers.GlobalAveragePooling1D()(embedding_sequence)
positive_probability = tf.keras.layers.Dense(units = 1, activation = 'sigmoid')(average_sequence)
model = tf.keras.Model(inputs = review_sequence, outputs = positive_probability)

In [39]:

model.summary()

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, None)]            0         
_________________________________________________________________
embedding_3 (Embedding)      (None, None, 16)          160016    
_________________________________________________________________
global_average_pooling1d_1 ( (None, 16)                0         
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 17        
=================================================================
Total params: 160,033
Trainable params: 160,033
Non-trainable params: 0
_________________________________________________________________

Compile, train, and evaluate the model¶

In [40]:

# Compile the model with a binary cross-entropy loss
model.compile(loss = 'binary_crossentropy', metrics = ['accuracy'], optimizer = 'adam')

In [41]:

# Train the model using .fit(), savng its history
history = model.fit (
    x_train, y_train, epochs = 5, batch_size = 32,
    validation_data = (x_test, y_test), validation_steps = 20
)

Epoch 1/5
782/782 [==============================] - 7s 8ms/step - loss: 0.6899 - accuracy: 0.5397 - val_loss: 0.6832 - val_accuracy: 0.5938
Epoch 2/5
782/782 [==============================] - 7s 8ms/step - loss: 0.6714 - accuracy: 0.6676 - val_loss: 0.6548 - val_accuracy: 0.6875
Epoch 3/5
782/782 [==============================] - 7s 8ms/step - loss: 0.6304 - accuracy: 0.7524 - val_loss: 0.6037 - val_accuracy: 0.7688
Epoch 4/5
782/782 [==============================] - 6s 7ms/step - loss: 0.5797 - accuracy: 0.7886 - val_loss: 0.5556 - val_accuracy: 0.7797
Epoch 5/5
782/782 [==============================] - 7s 8ms/step - loss: 0.5304 - accuracy: 0.8136 - val_loss: 0.5174 - val_accuracy: 0.7703

In [42]:

# Plot the training and validation accuracy
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')

history_dict = history.history

acc      = history_dict['accuracy']
val_acc  = history_dict['val_accuracy']
loss     = history_dict['loss']
val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

plt.figure(figsize=(14,5))
plt.plot(epochs, acc, marker = '.', label = 'Training acc')
plt.plot(epochs, val_acc, marker = '.', label = 'Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epoch')
plt.ylabel('Classification accuracy')
plt.legend(loc='lower right')
plt.ylim(0, 1)

Out[42]:

(0.0, 1.0)

The TensorFlow embedding projector¶

The Tensorflow embedding projector can be found here.

In [43]:

# Retrieve the embedding layer's weights from the trained model
weights = model.layers[1].get_weights()[0]

In [44]:

# Save the word Embeddings to tsv files
# Two files: 
#     one contains the embedding labels (meta.tsv),
#     one contains the embeddings (vecs.tsv)

import io
from os import path

out_v = io.open(path.join('data', 'vecs.tsv'), 'w', encoding = 'utf-8')
out_m = io.open(path.join('data', 'meta.tsv'), 'w', encoding = 'utf-8')

k = 0

for word, token in imdb_word_index.items():
    if k != 0:
        out_m.write('\n')
        out_v.write('\n')
    
    out_v.write('\t'.join([str(x) for x in weights[token]]))
    out_m.write(word)
    k += 1
    
out_v.close()
out_m.close()
# beware large collections of embeddings!

---

Recurrent neural network layers¶

Initialize and pass an input to a SimpleRNN layer¶

In [45]:

# Create a SimpleRNN layer and test it
simplernn_layer = tf.keras.layers.SimpleRNN(units = 16)

In [46]:

# Note that only the final cell output is returned
sequence = tf.constant([[[1., 1.], [2., 2.], [56., -100.]]])
layer_output = simplernn_layer(sequence)
layer_output

Out[46]:

<tf.Tensor: shape=(1, 16), dtype=float32, numpy=
array([[ 1.        ,  1.        ,  0.99982417,  1.        ,  1.        ,
         1.        , -1.        ,  1.        ,  0.9766807 , -1.        ,
         1.        ,  1.        ,  1.        ,  1.        ,  0.99653006,
        -1.        ]], dtype=float32)>

Load and transform the IMDB review sentiment dataset¶

In [47]:

# A function to load and preprocess the IMDB dataset
def get_and_pad_imdb_dataset(num_words = 10000, maxlen = None, index_from = 2):
    from tensorflow.keras.datasets import imdb

    # Load the reviews
    (x_train, y_train), (x_test, y_test) = imdb.load_data (
        path = 'imdb.npz', num_words = num_words, skip_top = 0, maxlen = maxlen,
        start_char = 1, oov_char = 2, index_from = index_from
    )

    x_train = tf.keras.preprocessing.sequence.pad_sequences (
        x_train, maxlen = None, padding = 'pre', truncating = 'pre', value = 0
    )
    
    x_test = tf.keras.preprocessing.sequence.pad_sequences (
        x_test, maxlen = None, padding = 'pre', truncating = 'pre', value = 0
    )

    return (x_train, y_train), (x_test, y_test)

In [48]:

# Load the dataset
(x_train, y_train), (x_test, y_test) = get_and_pad_imdb_dataset(maxlen = 250)

/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/imdb.py:155: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/imdb.py:156: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])

In [49]:

# A function to get the dataset word index
def get_imdb_word_index(num_words = 10000, index_from = 2):
    imdb_word_index = tf.keras.datasets.imdb.get_word_index(path = 'imdb_word_index.json')
    imdb_word_index = {
        key: value + index_from for key, value in imdb_word_index.items() if value <= num_words - index_from
    }
    return imdb_word_index

In [50]:

# Get the word index using get_imdb_word_index()
imdb_word_index = get_imdb_word_index()

Create a recurrent neural network model¶

In [51]:

# Get the maximum index value
max_index_value = max(imdb_word_index.values())
embedding_dim = 16

In [52]:

# Using Sequential, build the model:
# 1. Embedding.
# 2. LSTM.
# 3. Dense.
model = tf.keras.Sequential ([
    tf.keras.layers.Embedding(input_dim = max_index_value + 1, output_dim = embedding_dim, mask_zero = True),
    tf.keras.layers.LSTM(units = 16),
    tf.keras.layers.Dense(units = 1, activation = 'sigmoid'),
])

Compile and fit the model¶

In [53]:

# Compile the model with binary cross-entropy loss
model.compile(loss = 'binary_crossentropy', metrics = ['accuracy'], optimizer = 'adam')

In [54]:

# Fit the model and save its training history
history = model.fit(x_train, y_train, epochs = 3, batch_size = 32)

Epoch 1/3
782/782 [==============================] - 194s 248ms/step - loss: 0.3905 - accuracy: 0.8266
Epoch 2/3
782/782 [==============================] - 185s 237ms/step - loss: 0.2173 - accuracy: 0.9176
Epoch 3/3
782/782 [==============================] - 182s 233ms/step - loss: 0.1680 - accuracy: 0.9400

Plot learning curves¶

In [ ]:

# Plot the training and validation accuracy

import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')

history_dict = history.history

acc      = history_dict['accuracy']
val_acc  = history_dict['val_accuracy']
loss     = history_dict['loss']
val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

plt.figure(figsize=(14,5))
plt.plot(epochs, acc, marker='.', label='Training acc')
plt.plot(epochs, val_acc, marker='.', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epoch')
plt.ylabel('Classification accuracy')
plt.legend(loc='lower right')
plt.ylim(0, 1);

Make predictions with the model¶

In [56]:

# View the first test data example sentence
# (invert the word index)
inv_imdb_word_index = {value: key for key, value in imdb_word_index.items()}
print([inv_imdb_word_index[index] for index in x_test[0] if index > 2])

['ignore', 'the', 'bad', 'reviews', 'on', 'here', 'this', 'film', 'is', 'awesome', 'just', 'before', 'dawn', 'is', 'a', 'great', 'example', 'of', 'what', 'can', 'be', 'done', 'in', 'a', 'film', 'with', 'a', 'minimal', 'budget', 'if', 'you', 'have', 'a', 'dedicated', 'crew', 'decent', 'script', 'and', 'a', 'cool', 'idea', 'for', 'a', 'film', "it's", 'a', 'hell', 'of', 'a', 'lot', 'of', 'fun', 'br', 'br', 'i', 'enjoyed', 'it', 'a', 'lot', 'more', 'than', 'most', 'other', "80's", 'slashers', 'because', 'the', 'killer', 'is', 'so', 'unique', 'wrong', 'turn', 'ripped', 'this', 'movie', 'off', 'something', 'fierce', "there's", 'plenty', 'of', 'blood', 'and', 'scares', 'my', 'girlfriend', 'was', 'freaked', 'out', 'and', 'she', 'watches', 'almost', 'everything', 'with', 'me', 'and', "doesn't", "it's", 'got', 'that', 'creepiness', 'to', 'it', 'br', 'br', "i'd", 'say', 'that', 'just', 'before', 'dawn', 'is', 'the', 'best', 'early', "80's", 'slasher', 'out', 'there', 'i', 'really', 'enjoyed', 'it', 'br', 'br', '8', 'out', 'of', '10', 'kids']

In [57]:

# Get the model prediction using model.predict()
model.predict(x_test[None, 0, :])

Out[57]:

array([[0.9936923]], dtype=float32)

In [58]:

# Get the corresponding label
y_test[0]

Out[58]:

1

---

Stacked RNNs and the Bidirectional wrapper¶

Load and transform the IMDB review sentiment dataset¶

In [59]:

# A function to load and preprocess the IMDB dataset

def get_and_pad_imdb_dataset(num_words = 10000, maxlen = None, index_from = 2):
    from tensorflow.keras.datasets import imdb

    # Load the reviews
    (x_train, y_train), (x_test, y_test) = imdb.load_data (
        path = 'imdb.npz', num_words = num_words, skip_top = 0, maxlen = maxlen,
        start_char = 1, oov_char = 2, index_from = index_from
    )

    x_train = tf.keras.preprocessing.sequence.pad_sequences (
        x_train, maxlen = None, padding = 'pre', truncating = 'pre', value = 0
    )
    
    x_test = tf.keras.preprocessing.sequence.pad_sequences (
        x_test, maxlen = None, padding = 'pre', truncating = 'pre', value = 0
    )

    return (x_train, y_train), (x_test, y_test)

In [60]:

# Load the dataset
(x_train, y_train), (x_test, y_test) = get_and_pad_imdb_dataset(num_words = 5000, maxlen = 250)

/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/imdb.py:155: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/imdb.py:156: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])

In [61]:

# A function to get the dataset word index

def get_imdb_word_index(num_words = 10000, index_from = 2):
    imdb_word_index = tf.keras.datasets.imdb.get_word_index(path = 'imdb_word_index.json')
    imdb_word_index = {
        key: value + index_from for key, value in imdb_word_index.items() if value <= num_words - index_from
    }
    return imdb_word_index

In [62]:

# Get the word index using get_imdb_word_index()
imdb_word_index = get_imdb_word_index(num_words = 5000)

Build stacked and bidirectional recurrent models¶

In [63]:

# Get the maximum index value and specify an embedding dimension
max_index_value = max(imdb_word_index.values())
embedding_dim = 16

In [64]:

# Using Sequential, build a stacked LSTM model via return_sequences=True
model = tf.keras.Sequential ([
    tf.keras.layers.Embedding(input_dim = max_index_value + 1, output_dim = embedding_dim, mask_zero = True),
    tf.keras.layers.LSTM(units = 32, return_sequences = True),
    tf.keras.layers.LSTM(units = 32, return_sequences = False),
    tf.keras.layers.Dense(units = 1, activation = 'sigmoid'),
])

In [65]:

# Using Sequential, build a bidirectional RNN with merge_mode='sum'
model = tf.keras.Sequential ([
    tf.keras.layers.Embedding(input_dim = max_index_value + 1, output_dim = embedding_dim, mask_zero = True),
    tf.keras.layers.Bidirectional (
        layer = tf.keras.layers.LSTM(units = 8),
        merge_mode = 'sum',
        backward_layer = tf.keras.layers.GRU(units = 8, go_backwards = True),
    ),
    tf.keras.layers.Dense(units = 1, activation = 'sigmoid'),
])

In [66]:

# Create a model featuring both stacked recurrent layers and a bidirectional layer
model = tf.keras.Sequential ([
    tf.keras.layers.Embedding(input_dim = max_index_value + 1, output_dim = embedding_dim),
    tf.keras.layers.Bidirectional (
        layer = tf.keras.layers.LSTM(units = 8, return_sequences = True),
        merge_mode = 'concat'
    ),
    tf.keras.layers.GRU(units = 8, return_sequences = False),
    tf.keras.layers.Dense(units = 1, activation = 'sigmoid'),
])

Compile and fit the model¶

In [67]:

# Compile the model
model.compile(loss = 'binary_crossentropy', metrics = ['accuracy'], optimizer = 'adam')

In [68]:

# Train the model, saving its history
history = model.fit(x_train, y_train, epochs = 3, batch_size = 32)

Epoch 1/3
782/782 [==============================] - 304s 388ms/step - loss: 0.4000 - accuracy: 0.8100
Epoch 2/3
782/782 [==============================] - 302s 386ms/step - loss: 0.2494 - accuracy: 0.9044
Epoch 3/3
782/782 [==============================] - 291s 372ms/step - loss: 0.1967 - accuracy: 0.9257

In [ ]:

# Plot the training and validation accuracy

import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')

history_dict = history.history

acc      = history_dict['accuracy']
val_acc  = history_dict['val_accuracy']
loss     = history_dict['loss']
val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

plt.figure(figsize=(14,5))
plt.plot(epochs, acc, marker='.', label='Training acc')
plt.plot(epochs, val_acc, marker='.', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epoch')
plt.ylabel('Classification accuracy')
plt.legend(loc='lower right')
plt.ylim(0, 1);

The Build Method

In this reading you will learn how to use the build method to allow custom layers to work with flexible sized inputs.

In [1]:

import tensorflow as tf
print(tf.__version__)

2.2.0

Fix the input shape in the custom layer¶

Previously, you have created custom layers by initialising all variables in the __init__ method. For instance, you defined a dense layer called MyLayer as follows:

In [2]:

# Create a custom layer

from tensorflow.keras.layers import Layer

class MyLayer(Layer):

    def __init__(self, units, input_dim, **kwargs):
        super(MyLayer, self).__init__(**kwargs)
        self.w = self.add_weight(shape=(input_dim, units),
                             initializer='random_normal')
        self.b = self.add_weight(shape=(units,),
                             initializer='zeros')
        
    def call(self, inputs):
        return tf.matmul(inputs, self.w)+self.b

Notice that the required arguments for the __init__ method are the number of units in the dense layer (units) and the input size (input_dim). This means that you need to fix these two arguments when you instantiate the layer.

In [3]:

#  Create a custom layer with 3 units and input dimension of 5

dense_layer = MyLayer(3, 5) 

Since the input size has been fixed to be 5, this custom layer can only take inputs of that size. For example, we can call the layer as follows:

In [4]:

# Call the custom layer on a Tensor input of ones

x = tf.ones((1,5))
print(dense_layer(x))

tf.Tensor([[-0.15427749  0.32772803  0.1284101 ]], shape=(1, 3), dtype=float32)

However, forcing the input shape (and therefore the shape of the weights) to be fixed when the layer is instantiated is unnecessary, and it may be more convenient to only do this later on, after the model has been defined.

For example, in some cases you may not know the input shape at the model building time. We have come across this concept before when building models with the Sequential API. If the input_shape argument is omitted, the weights will only be created when an input is passed into the model.

Allow a flexible input shape in the custom layer¶

You can delay the weight creation by using the build method to define the weights. The build method is executed when the __call__ method is called, meaning the weights are only created only the layer is called with a specific input.

The build method has a required argument input_shape, which can be used to define the shapes of the layer weights.

In [5]:

# Rewrite the custom layer with lazy weight creation

class MyLayer(Layer):

    def __init__(self, units, **kwargs):
        super(MyLayer, self).__init__(**kwargs)
        self.units = units
    
    def build(self, input_shape):
        self.w = self.add_weight(shape=(input_shape[-1], self.units),
                                 initializer='random_normal')
        self.b = self.add_weight(shape=(self.units,),
                                 initializer='zeros')
    def call(self, inputs):
        return tf.matmul(inputs, self.w)+self.b

Now, when you instantiate the layer, you only need to specify the number of units in the dense layer (units), and not the input size (input_dim).

Create a custom layer with flexible input size¶

In [6]:

#  Create a custom layer with 3 units

dense_layer = MyLayer(3) 

This layer can now be called on an input of any size, at which point the layer weights will be created and the input size will be fixed.

In [7]:

# Call the custom layer on a Tensor input of ones of size 5

x = tf.ones((1,5))
print(dense_layer(x))

tf.Tensor([[-0.10292004  0.02630131  0.13124558]], shape=(1, 3), dtype=float32)

In [8]:

# Print the layer weights

dense_layer.weights

Out[8]:

[<tf.Variable 'my_layer_1/Variable:0' shape=(5, 3) dtype=float32, numpy=
 array([[-0.09217734, -0.02080498,  0.06012938],
        [-0.03796877,  0.04266685,  0.02407069],
        [-0.00704942,  0.03004533,  0.01339882],
        [-0.0033752 , -0.03693453,  0.03387153],
        [ 0.03765069,  0.01132864, -0.00022485]], dtype=float32)>,
 <tf.Variable 'my_layer_1/Variable:0' shape=(3,) dtype=float32, numpy=array([0., 0., 0.], dtype=float32)>]

Create a new custom layer and pass in a different sized input¶

In [9]:

#  Create a new custom layer with 3 units

dense_layer = MyLayer(3) 

In [10]:

# Call the custom layer on a Tensor input of ones of size 4

x = tf.ones((1,4))
print(dense_layer(x))

tf.Tensor([[-0.11568737  0.13634682 -0.06427702]], shape=(1, 3), dtype=float32)

In [11]:

# Print the layer weights

dense_layer.weights

Out[11]:

[<tf.Variable 'my_layer_2/Variable:0' shape=(4, 3) dtype=float32, numpy=
 array([[-0.02903587,  0.04687321,  0.00941058],
        [-0.01757504,  0.01653846, -0.0232079 ],
        [ 0.00094642, -0.0014191 , -0.02150488],
        [-0.07002288,  0.07435425, -0.02897482]], dtype=float32)>,
 <tf.Variable 'my_layer_2/Variable:0' shape=(3,) dtype=float32, numpy=array([0., 0., 0.], dtype=float32)>]

Note that the code for creating a custom layer object is identical, but the shape of the weights in the layer depend on the size of the input passed to the layer.

Flexible input shapes in models¶

Deferring the weight creation until the layer is called is also useful when using the custom layer as an intermediate layer inside a larger model. In this case you may want to create several custom layer objects in the model, and it is tedious to keep track of the input shape that each of the custom layers needs.

By deferring the weight creation as above, the input shape can be inferred from the output of the previous layer.

In [12]:

# Create a model using the custom layer

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Softmax

class MyModel(Model):

    def __init__(self, units_1, units_2, **kwargs):
        super(MyModel, self).__init__(**kwargs)
        self.layer_1 = MyLayer(units_1)
        self.layer_2 = MyLayer(units_2)

    def call(self, inputs):
        x = self.layer_1(inputs)
        x = tf.nn.relu(x)
        x = self.layer_2(x)
        return Softmax()(x)

In the above model definition, the custom layer MyLayer is used twice. Notice that each instance of the custom layer object can have a different input size, depending on the arguments used to create the model and the inputs passed into the model

In [13]:

# Create a custom model object

model = MyModel(units_1=32, units_2=10)

We can create and initialise all of the weights of the model by passing in an example Tensor input.

In [14]:

# Create and initialize all of the model weights

_ = model(tf.ones((1, 100)))

In [15]:

# Print the model summary

model.summary()

Model: "my_model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
my_layer_3 (MyLayer)         multiple                  3232      
_________________________________________________________________
my_layer_4 (MyLayer)         multiple                  330       
=================================================================
Total params: 3,562
Trainable params: 3,562
Non-trainable params: 0
_________________________________________________________________

Further reading and resources¶

• https://www.tensorflow.org/guide/keras/custom_layers_and_models#best_practice_deferring_weight_creation_until_the_shape_of_the_inputs_is_known

Tracking Metrics In Custom Training Loops

In this reading you will learn how to track metrics such as accuracy within custom training loops.

In [1]:

import tensorflow as tf
print(tf.__version__)

2.2.0

Tracking performance metrics is useful to check the performance of your model during and after training. Previously, we have passed in a list of metrics to the model.fit or model.evaluate methods when using the higher level APIs. However, we would still like to be able to evaluate our model on performance metrics in a custom training loop.

Load the dataset¶

For this reading notebook we will use the Reuters dataset. The cells below load and process the dataset, and prepare it as tf.data.Dataset objects.

In [2]:

# Load the dataset

from tensorflow.keras.datasets import reuters

(train_data, train_labels), (test_data, test_labels) = reuters.load_data(num_words=10000, skip_top=50)

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/reuters.npz
2113536/2110848 [==============================] - 3s 1us/step

/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/reuters.py:148: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/reuters.py:149: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])

In [3]:

# Define the class names

class_names = ['cocoa','grain','veg-oil','earn','acq','wheat','copper','housing','money-supply',
   'coffee','sugar','trade','reserves','ship','cotton','carcass','crude','nat-gas',
   'cpi','money-fx','interest','gnp','meal-feed','alum','oilseed','gold','tin',
   'strategic-metal','livestock','retail','ipi','iron-steel','rubber','heat','jobs',
   'lei','bop','zinc','orange','pet-chem','dlr','gas','silver','wpi','hog','lead']

In [4]:

# Pad and truncate the train and test inputs

from tensorflow.keras.preprocessing.sequence import pad_sequences

padded_train_data = pad_sequences(train_data, maxlen=100, truncating='post')
padded_test_data = pad_sequences(test_data, maxlen=100, truncating='post')

In [5]:

# Split the training data into training and validation sets

from sklearn.model_selection import train_test_split

train_data, val_data, train_labels, val_labels = train_test_split(padded_train_data, train_labels, 
                                                                  test_size=0.3)

In [6]:

# Create Datasets for training, validation and test data

train_dataset = tf.data.Dataset.from_tensor_slices((train_data, train_labels))
train_dataset = train_dataset.batch(32)

val_dataset = tf.data.Dataset.from_tensor_slices((val_data, val_labels))
val_dataset = val_dataset.shuffle(500)
val_dataset = val_dataset.batch(32)

test_dataset = tf.data.Dataset.from_tensor_slices((padded_test_data, test_labels))
test_dataset = test_dataset.batch(32)

Define the model, optimizer and loss function¶

Before getting to the custom training loop, we first define the model. We will use the model subclassing API to build a recurrent neural network for this example.

In [7]:

# Build the model

from tensorflow.keras.layers import Embedding, GRU, Bidirectional, Dense
from tensorflow.keras.models import Model

class RNNModel(Model):

    def __init__(self, units_1, units_2, num_classes, **kwargs):
        super(RNNModel, self).__init__(**kwargs)
        self.embedding = Embedding(input_dim=10000, output_dim=16, input_length=100)
        self.gru_1 = Bidirectional(GRU(units_1, return_sequences=True), merge_mode='sum')
        self.gru_2 = GRU(units_2)
        self.dense = Dense(num_classes, activation='softmax')

    def call(self, inputs):
        h = self.embedding(inputs)
        h = self.gru_1(h)
        h = self.gru_2(h)
        return self.dense(h)

In [8]:

# Instantiate the model

model = RNNModel(units_1=32, units_2=16, num_classes=46, name='rnn_model')

We will use the SGD optimizer with Nesterov momentum, and a learning rate of 0.005:

In [9]:

# Define the optimizer

optimizer = tf.keras.optimizers.SGD(learning_rate=0.005, momentum=0.9, nesterov=True)

The loss function for our model should be the sparse categorical cross entropy, since the labels are in sparse form:

In [10]:

# Define the loss

loss = tf.keras.losses.SparseCategoricalCrossentropy()

Build the custom training loop¶

The following function computes the model predictions, loss and gradients, for use in the custom training loop.

In [11]:

# Define a function to compute the loss and gradients

def grad(model, inputs, targets, loss):
    with tf.GradientTape() as tape:
        preds = model(inputs)
        loss_value = loss(targets, preds)
    return preds, loss_value, tape.gradient(loss_value, model.trainable_variables)

Metric objects can be created and used to track performance measures in the custom training loop. We will set up our custom training loop to track the average loss, and area under the ROC curve (ROC AUC). Of course there are many more metrics that you could use.

In [12]:

# Create lists to accumulate the metric values

train_loss_results = []
train_roc_auc_results = []

val_loss_results = []
val_roc_auc_results = []

In the following custom training loop, we define an outer loop for the epochs, and an inner loop for the batches in the training dataset. At the end of each epoch we run a validation loop for a number of iterations.

Inside the inner loop we use the metric objects to calculate the metric evaluation values. These values are then appended to the empty lists. The metric objects are re-initialised at the start of each epoch.

The following training loop may take a while to run. Feel free to reduce the number of epochs if you want.

In [13]:

# Run the custom training loop

from tensorflow.keras.utils import to_categorical

num_epochs = 5
val_steps = 10

for epoch in range(num_epochs):
    train_epoch_loss_avg = tf.keras.metrics.Mean()
    train_epoch_roc_auc = tf.keras.metrics.AUC(curve='ROC')
    
    val_epoch_loss_avg = tf.keras.metrics.Mean()
    val_epoch_roc_auc = tf.keras.metrics.AUC(curve='ROC')

    for inputs, labels in train_dataset:
        model_preds, loss_value, grads = grad(model, inputs, labels, loss)
        optimizer.apply_gradients(zip(grads, model.trainable_variables))

        train_epoch_loss_avg(loss_value)
        train_epoch_roc_auc(to_categorical(labels, num_classes=46), model_preds)
        
    for inputs, labels in val_dataset.take(val_steps):
        model_preds = model(inputs)
        val_epoch_loss_avg(loss(labels, model_preds))
        val_epoch_roc_auc(to_categorical(labels, num_classes=46), model_preds)
        
    train_loss_results.append(train_epoch_loss_avg.result().numpy())
    train_roc_auc_results.append(train_epoch_roc_auc.result().numpy())
    
    val_loss_results.append(val_epoch_loss_avg.result().numpy())
    val_roc_auc_results.append(val_epoch_roc_auc.result().numpy())

    print("Epoch {:03d}: Training loss: {:.3f}, ROC AUC: {:.3%}".format(epoch, train_epoch_loss_avg.result(), 
                                                                        train_epoch_roc_auc.result()))
    print("              Validation loss: {:.3f}, ROC AUC {:.3%}".format(val_epoch_loss_avg.result(),
                                                                         val_epoch_roc_auc.result()))

Epoch 000: Training loss: 2.744, ROC AUC: 84.445%
              Validation loss: 2.385, ROC AUC 89.292%
Epoch 001: Training loss: 2.417, ROC AUC: 88.517%
              Validation loss: 2.401, ROC AUC 89.397%
Epoch 002: Training loss: 2.396, ROC AUC: 89.044%
              Validation loss: 2.411, ROC AUC 89.823%
Epoch 003: Training loss: 2.384, ROC AUC: 89.283%
              Validation loss: 2.352, ROC AUC 89.845%
Epoch 004: Training loss: 2.369, ROC AUC: 89.384%
              Validation loss: 2.348, ROC AUC 90.220%

Plot the learning curves¶

In [14]:

import matplotlib.pyplot as plt

fig = plt.figure(figsize=(15, 5))

fig.add_subplot(121)
plt.plot(train_loss_results)
plt.plot(val_loss_results)
plt.title('Loss vs. epochs')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Training', 'Validation'], loc='upper right')

fig.add_subplot(122)
plt.plot(train_roc_auc_results)
plt.plot(val_roc_auc_results)
plt.title('ROC AUC vs. epochs')
plt.ylabel('ROC AUC')
plt.xlabel('Epoch')
plt.legend(['Training', 'Validation'], loc='lower right')
plt.show() 

Test the model¶

Similarly, we can calculate the loss and metric values on the test dataset.

In [15]:

# Create metric objects for testing

test_epoch_loss_avg = tf.keras.metrics.Mean()
test_epoch_roc_auc = tf.keras.metrics.AUC(curve='ROC')

In [16]:

# Calculate metrics on the test set

for inputs, labels in test_dataset:
    model_pred = model(inputs)
    test_epoch_loss_avg(loss(labels, model_pred))  
    test_epoch_roc_auc(to_categorical(labels, num_classes=46), model_pred)

print("Test loss: {:.3f}".format(test_epoch_loss_avg.result().numpy()))
print("Test ROC AUC: {:.3%}".format(test_epoch_roc_auc.result().numpy()))

Test loss: 2.380
Test ROC AUC: 89.125%

Further reading and resources¶

• https://www.tensorflow.org/tutorials/customization/custom_training_walkthrough#training_loop
• https://www.tensorflow.org/api_docs/python/tf/keras/metrics
• https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve

Model Subclassing And Custom Training Loops

In [1]:

import tensorflow as tf
print(tf.__version__)

2.2.0

1. Model subclassing¶

2. Custom layers¶

3. Automatic differentiation¶

4. Custom training loops¶

5. tf.function decorator¶

---

Model subclassing¶

In [2]:

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, Dropout, Softmax, concatenate

Create a simple model using the model subclassing API¶

In [3]:

# Build the model

class MyModel(Model):

    def __init__(self):
        super(MyModel, self).__init__()
        self.dense_1 = Dense(64, activation = 'relu')
        self.dense_2 = Dense(10)
        self.dense_3 = Dense(5)
        self.softmax = Softmax()

    def call(self, inputs):
        x = self.dense_1(inputs)
        y1 = self.dense_2(inputs)
        y2 = self.dense_3(y1)
        concat = concatenate([x, y2])
        return self.softmax(concat)

In [4]:

# Print the model summary
model = MyModel()
model(tf.random.uniform([1, 10]))
model.summary()

Model: "my_model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                multiple                  704       
_________________________________________________________________
dense_1 (Dense)              multiple                  110       
_________________________________________________________________
dense_2 (Dense)              multiple                  55        
_________________________________________________________________
softmax (Softmax)            multiple                  0         
=================================================================
Total params: 869
Trainable params: 869
Non-trainable params: 0
_________________________________________________________________

---

Custom layers¶

In [5]:

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Layer, Softmax

Create custom layers¶

In [6]:

# Create a custom layer

class MyLayer(Layer):
    
    def __init__(self, units, input_dim):
        super(MyLayer, self).__init__()
        self.w = self.add_weight(shape = (input_dim, units), initializer = 'random_normal')
        self.b = self.add_weight(shape = (units,), initializer = 'zeros')
        
    def call(self, inputs):
        return tf.matmul(inputs, self.w) + self.b
    
dense_layer = MyLayer(3, 5)
x = tf.ones((1, 5))
print(dense_layer(x))
print(dense_layer.weights)

tf.Tensor([[ 0.13339971 -0.00786851  0.11008401]], shape=(1, 3), dtype=float32)
[<tf.Variable 'Variable:0' shape=(5, 3) dtype=float32, numpy=
array([[ 0.06737228, -0.03497477,  0.03775391],
       [ 0.04375362, -0.01718085, -0.01942822],
       [ 0.05162938, -0.04853738,  0.07312917],
       [ 0.00101959,  0.07783843,  0.0581594 ],
       [-0.03037515,  0.01498607, -0.03953025]], dtype=float32)>, <tf.Variable 'Variable:0' shape=(3,) dtype=float32, numpy=array([0., 0., 0.], dtype=float32)>]

In [7]:

# Specify trainable weights

class MyLayer(Layer):
    
    def __init__(self, units, input_dim):
        super(MyLayer, self).__init__()
        self.w = self.add_weight(shape = (input_dim, units), initializer = 'random_normal', trainable = False)
        self.b = self.add_weight(shape = (units,), initializer = 'zeros', trainable = False)
        
    def call(self, inputs):
        return tf.matmul(inputs, self.w) + self.b
    
dense_layer = MyLayer(3, 5)

In [8]:

print('trainable weights:', len(dense_layer.trainable_weights))
print('non-trainable weights:', len(dense_layer.non_trainable_weights))

trainable weights: 0
non-trainable weights: 2

In [9]:

# Create a custom layer to accumulate means of output values

class MyLayerMean(Layer):
    
    def __init__(self, units, input_dim):
        super(MyLayerMean, self).__init__()
        self.w = self.add_weight(shape = (input_dim, units), initializer = 'random_normal')
        self.b = self.add_weight(shape = (units,), initializer = 'zeros')
        self.sum_activation = tf.Variable(initial_value = tf.zeros((units,)), trainable = False)
        self.number_call = tf.Variable(initial_value = 0, trainable = False)
        
    def call(self, inputs):
        activation = tf.matmul(inputs, self.w) + self.b
        self.sum_activation.assign_add(tf.reduce_sum(activation, axis = 0))
        self.number_call.assign_add(inputs.shape[0])
        return activation, self.sum_activation / tf.cast(self.number_call, tf.float32)

dense_layer = MyLayerMean(3, 5)

In [10]:

# Test the layer

y, activation_means = dense_layer(tf.ones((1, 5)))
print(activation_means.numpy())

y, activation_means = dense_layer(tf.ones((1, 5)))
print(activation_means.numpy())

[ 0.32861292  0.09184293 -0.04379604]
[ 0.32861292  0.09184293 -0.04379604]

In [11]:

# Create a Dropout layer as a custom layer

class MyDropout(Layer):

    def __init__(self, rate):
        super(MyDropout, self).__init__()
        self.rate = rate
        
    def call(self, inputs):
        # Define forward pass for dropout layer
        return tf.nn.dropout(inputs, rate = self.rate)

Implement the custom layers into a model¶

In [12]:

# Build the model using custom layers with the model subclassing API

class MyModel(Model):

    def __init__(self, units_1, input_dim_1, units_2, units_3):
        super(MyModel, self).__init__()
        # Define layers
        self.layer_1 = MyLayer(units_1, input_dim_1)
        self.dropout_1 = MyDropout(0.5)
        self.layer_2 = MyLayer(units_2, units_1)
        self.dropout_2 = MyDropout(0.5)
        self.layer_3 = MyLayer(units_3, units_2)
        self.softmax = Softmax()

    def call(self, inputs):
        # Define forward pass
        x = self.layer_1(inputs)
        x = tf.nn.relu(x)
        x = self.dropout_1(x)
        x = self.layer_2(x)
        x = tf.nn.relu(x)
        x = self.dropout_2(x)
        x = self.layer_3(x)
        return self.softmax(x)

In [13]:

# Instantiate a model object

model = MyModel(64, 10000, 64, 46)
print(model(tf.ones((1, 10000))))
model.summary()

tf.Tensor(
[[0.03710695 0.00589327 0.00708813 0.02279445 0.03346972 0.00655904
  0.00579258 0.0369241  0.00654865 0.01231807 0.00345892 0.05747055
  0.00519882 0.00400936 0.0608132  0.0054371  0.02344397 0.00704151
  0.01509309 0.03241989 0.01107135 0.01149343 0.03617447 0.00221361
  0.01899736 0.01680321 0.01086698 0.03196133 0.05563021 0.0347878
  0.00740595 0.09717432 0.00365099 0.04109014 0.01600296 0.01282467
  0.01123402 0.03369091 0.01771386 0.00522895 0.00987317 0.01434885
  0.01963162 0.01411207 0.05191726 0.02521919]], shape=(1, 46), dtype=float32)
Model: "my_model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
my_layer_2 (MyLayer)         multiple                  640064    
_________________________________________________________________
my_dropout (MyDropout)       multiple                  0         
_________________________________________________________________
my_layer_3 (MyLayer)         multiple                  4160      
_________________________________________________________________
my_dropout_1 (MyDropout)     multiple                  0         
_________________________________________________________________
my_layer_4 (MyLayer)         multiple                  2990      
_________________________________________________________________
softmax_1 (Softmax)          multiple                  0         
=================================================================
Total params: 647,214
Trainable params: 0
Non-trainable params: 647,214
_________________________________________________________________

---

Automatic differentiation¶

In [14]:

import numpy as np
import matplotlib.pyplot as plt

Create synthetic data¶

In [15]:

# Create data from a noise contaminated linear model

def MakeNoisyData(m, b, n = 20):
    x = tf.random.uniform(shape = (n,))
    noise = tf.random.normal(shape = (len(x),), stddev = 0.1)
    y = m * x + b + noise
    return x, y

m = 1
b = 2
x_train, y_train = MakeNoisyData(m, b)
plt.plot(x_train, y_train, 'b.')

Out[15]:

[<matplotlib.lines.Line2D at 0x7f4e3c4dcf10>]

Define a linear regression model¶

In [16]:

from tensorflow.keras.layers import Layer

In [17]:

# Build a custom layer for the linear regression model

class LinearLayer(Layer):

    def __init__(self):
        super(LinearLayer, self).__init__()
        self.m = self.add_weight(shape = (1,), initializer = 'random_normal')
        self.b = self.add_weight(shape = (1,), initializer = 'zeros')
        
    def call(self, inputs):
        return self.m * inputs + self.b
    
linear_regression = LinearLayer()

print(linear_regression(x_train))
print(linear_regression.weights)

tf.Tensor(
[-1.7461871e-03 -3.7088564e-03 -2.8419413e-03 -3.1810191e-03
 -5.2967407e-03 -4.8893965e-03 -4.5748414e-03 -1.7225604e-04
 -6.8192319e-03 -2.9910973e-03 -1.3603655e-03 -3.8445042e-03
 -4.1214572e-03 -7.7773537e-04 -5.0585240e-05 -6.1724773e-03
 -5.7818606e-03 -4.7573359e-03 -5.1108198e-03 -4.6941526e-03], shape=(20,), dtype=float32)
[<tf.Variable 'Variable:0' shape=(1,) dtype=float32, numpy=array([-0.00748897], dtype=float32)>, <tf.Variable 'Variable:0' shape=(1,) dtype=float32, numpy=array([0.], dtype=float32)>]

Define the loss function¶

In [18]:

# Define the mean squared error loss function

def SquaredError(y_pred, y_true):
    return tf.reduce_mean(tf.square(y_pred - y_true)) 

starting_loss = SquaredError(linear_regression(x_train), y_train)
print("Starting loss", starting_loss.numpy())

Starting loss 6.419444

Train and plot the model¶

In [19]:

# Implement a gradient descent training loop for the linear regression model

learning_rate = 0.25
steps = 25

for i in range(steps):
    
    with tf.GradientTape() as tape:
        predictions = linear_regression(x_train)
        loss = SquaredError(predictions, y_train)
        
    gradients = tape.gradient(loss, linear_regression.trainable_variables)
    
    linear_regression.m.assign_sub(learning_rate * gradients[0])
    linear_regression.b.assign_sub(learning_rate * gradients[1])
    
    print('Step %d, Loss %f' % (i, loss.numpy()))

Step 0, Loss 6.419444
Step 1, Loss 0.909666
Step 2, Loss 0.135023
Step 3, Loss 0.026103
Step 4, Loss 0.010780
Step 5, Loss 0.008616
Step 6, Loss 0.008303
Step 7, Loss 0.008251
Step 8, Loss 0.008235
Step 9, Loss 0.008226
Step 10, Loss 0.008217
Step 11, Loss 0.008209
Step 12, Loss 0.008201
Step 13, Loss 0.008194
Step 14, Loss 0.008187
Step 15, Loss 0.008181
Step 16, Loss 0.008175
Step 17, Loss 0.008169
Step 18, Loss 0.008163
Step 19, Loss 0.008158
Step 20, Loss 0.008153
Step 21, Loss 0.008148
Step 22, Loss 0.008144
Step 23, Loss 0.008140
Step 24, Loss 0.008136

In [20]:

# Plot the learned regression model

print("m:{},  trained m:{}".format(m,linear_regression.m.numpy()))
print("b:{},  trained b:{}".format(b,linear_regression.b.numpy()))

plt.plot(x_train, y_train, 'b.')

x_linear_regression=np.linspace(min(x_train), max(x_train),50)
plt.plot(x_linear_regression, linear_regression.m*x_linear_regression+linear_regression.b, 'r.')

m:1,  trained m:[0.9969611]
b:2,  trained b:[2.0299757]

Out[20]:

[<matplotlib.lines.Line2D at 0x7f4e6c1222d0>]

---

Custom training loops¶

In [21]:

import numpy as np
import matplotlib.pyplot as plt
import time

Build the model¶

In [22]:

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Layer, Softmax

In [23]:

# Define the custom layers and model

class MyLayer(Layer):
    
    def __init__(self, units):
        super(MyLayer, self).__init__()
        self.units = units
        
    def build(self, input_shape):
        self.w = self.add_weight(shape = (input_shape[-1], self.units), initializer = 'random_normal', name = 'kernel')
        self.b = self.add_weight(shape = (self.units,), initializer = 'zeros', name = 'bias')
        
    def call(self, inputs):
        return tf.matmul(inputs, self.w) + self.b
    
class MyDropout(Layer):

    def __init__(self, rate):
        super(MyDropout, self).__init__()
        self.rate = rate
        
    def call(self, inputs):
        # Define forward pass for dropout layer
        return tf.nn.dropout(inputs, rate = self.rate)
    
class MyModel(Model):

    def __init__(self, units_1, units_2, units_3):
        super(MyModel, self).__init__()
        # Define layers
        self.layer_1 = MyLayer(units_1)
        self.dropout_1 = MyDropout(0.5)
        self.layer_2 = MyLayer(units_2)
        self.dropout_2 = MyDropout(0.5)
        self.layer_3 = MyLayer(units_3)
        self.softmax = Softmax()

    def call(self, inputs):
        # Define forward pass
        x = self.layer_1(inputs)
        x = tf.nn.relu(x)
        x = self.dropout_1(x)
        x = self.layer_2(x)
        x = tf.nn.relu(x)
        x = self.dropout_2(x)
        x = self.layer_3(x)
        return self.softmax(x)

In [24]:

# Instantiate the model object
model = MyModel(64, 64, 46)
print(model(tf.ones((1, 10000))))
model.summary()

tf.Tensor(
[[0.01089813 0.01250871 0.02539792 0.01158731 0.03327763 0.0217747
  0.0138029  0.01765059 0.02002498 0.01965816 0.02154693 0.01684865
  0.02405405 0.02545902 0.01735986 0.00877715 0.02755862 0.02383961
  0.04849238 0.02666703 0.02129876 0.02868291 0.05854288 0.01882431
  0.0139344  0.01409555 0.0279922  0.0199068  0.03093504 0.01712411
  0.02680226 0.01971242 0.00526185 0.0108718  0.00617074 0.01402272
  0.01169078 0.01620276 0.03181221 0.01706836 0.01832475 0.00852291
  0.04197437 0.02737742 0.04744845 0.01821483]], shape=(1, 46), dtype=float32)
Model: "my_model_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
my_layer_5 (MyLayer)         multiple                  640064    
_________________________________________________________________
my_dropout_2 (MyDropout)     multiple                  0         
_________________________________________________________________
my_layer_6 (MyLayer)         multiple                  4160      
_________________________________________________________________
my_dropout_3 (MyDropout)     multiple                  0         
_________________________________________________________________
my_layer_7 (MyLayer)         multiple                  2990      
_________________________________________________________________
softmax_2 (Softmax)          multiple                  0         
=================================================================
Total params: 647,214
Trainable params: 647,214
Non-trainable params: 0
_________________________________________________________________

Load the reuters dataset and define the class_names¶

In [25]:

# Load the dataset

from tensorflow.keras.datasets import reuters

(train_data, train_labels), (test_data, test_labels) = reuters.load_data(num_words = 10000)

class_names = [
    'cocoa','grain','veg-oil','earn','acq','wheat','copper','housing','money-supply',
    'coffee','sugar','trade','reserves','ship','cotton','carcass','crude','nat-gas',
    'cpi','money-fx','interest','gnp','meal-feed','alum','oilseed','gold','tin',
    'strategic-metal','livestock','retail','ipi','iron-steel','rubber','heat','jobs',
    'lei','bop','zinc','orange','pet-chem','dlr','gas','silver','wpi','hog','lead'
]

/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/reuters.py:148: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
/home/bacti/anaconda3/envs/tensor/lib/python3.7/site-packages/tensorflow/python/keras/datasets/reuters.py:149: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])

In [26]:

# Print the class of the first sample
print("Label: {}".format(class_names[train_labels[0]]))

Label: earn

Get the dataset word index¶

In [27]:

# Load the Reuters word index

word_to_index = reuters.get_word_index()

invert_word_index = dict([(value, key) for (key, value) in word_to_index.items()])
text_news = ' '.join([invert_word_index.get(i - 3, '?') for i in train_data[0]])

In [28]:

# Print the first data example sentence
print(text_news)

? ? ? said as a result of its december acquisition of space co it expects earnings per share in 1987 of 1 15 to 1 30 dlrs per share up from 70 cts in 1986 the company said pretax net should rise to nine to 10 mln dlrs from six mln dlrs in 1986 and rental operation revenues to 19 to 22 mln dlrs from 12 5 mln dlrs it said cash flow per share this year should be 2 50 to three dlrs reuter 3

Preprocess the data¶

In [29]:

# Define a function that encodes the data into a 'bag of words' representation

def bag_of_words(text_samples, elements = 10000):
    output = np.zeros((len(text_samples), elements))
    for i, word in enumerate(text_samples):
        output[i, word] = 1.
    return output

x_train = bag_of_words(train_data)
x_test = bag_of_words(test_data)

print("Shape of x_train:", x_train.shape)
print("Shape of x_test:", x_test.shape)

Shape of x_train: (8982, 10000)
Shape of x_test: (2246, 10000)

Define the loss function and optimizer¶

In [30]:

# Define the categorical cross entropy loss and Adam optimizer

loss_object = tf.keras.losses.SparseCategoricalCrossentropy()

def loss(model, x, y, wd):
    kernel_variables = []
    for l in model.layers:
        for w in l.weights:
            if 'kernel' in w.name:
                kernel_variables.append(w)
    wd_penalty = wd * tf.reduce_sum([tf.reduce_sum(tf.square(k)) for k in kernel_variables])
    y_ = model(x)
    return loss_object(y_true = y, y_pred = y_) + wd_penalty

optimizer = tf.keras.optimizers.Adam(learning_rate = 0.001)

Train the model¶

In [31]:

# Define a function to compute the forward and backward pass

def grad(model, inputs, targets, wd):
    with tf.GradientTape() as tape:
        loss_value = loss(model, inputs, targets, wd)
    return loss_value, tape.gradient(loss_value, model.trainable_variables)

In [32]:

# Implement the training loop

from tensorflow.keras.utils import to_categorical

start_time = time.time()

train_dataset = tf.data.Dataset.from_tensor_slices((x_train, train_labels))
train_dataset = train_dataset.batch(32)

# Keep results for plotting
train_loss_results = []
train_accuracy_results = []

num_epochs = 10
weight_decay = 0.005

for epoch in range(num_epochs):
    epoch_loss_avg = tf.keras.metrics.Mean()
    epoch_accuracy = tf.keras.metrics.CategoricalAccuracy()
    
    # Training loop
    for x, y in train_dataset:
        # Optimize the model
        loss_value, grads = grad(model, x, y, weight_decay)
        optimizer.apply_gradients(zip(grads, model.trainable_variables))
        
        # Compute current loop
        epoch_loss_avg(loss_value)
        # Compare predicted label to actual label
        epoch_accuracy(to_categorical(y), model(x))
        
    # End epoch
    train_loss_results.append(epoch_loss_avg.result())
    train_accuracy_results.append(epoch_accuracy.result())
    
    print('Epoch {:03d}: Loss {:.3f}, Accuracy: {:.3%}'.format(epoch, epoch_loss_avg.result(), epoch_accuracy.result()))
    
print("Duration :{:.3f}".format(time.time() - start_time))

WARNING:tensorflow:Layer my_model_2 is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.

If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

Epoch 000: Loss 3.302, Accuracy: 49.176%
Epoch 001: Loss 1.911, Accuracy: 60.543%
Epoch 002: Loss 1.840, Accuracy: 64.485%
Epoch 003: Loss 1.799, Accuracy: 67.234%
Epoch 004: Loss 1.752, Accuracy: 68.771%
Epoch 005: Loss 1.731, Accuracy: 69.584%
Epoch 006: Loss 1.733, Accuracy: 69.595%
Epoch 007: Loss 1.717, Accuracy: 70.174%
Epoch 008: Loss 1.705, Accuracy: 70.341%
Epoch 009: Loss 1.710, Accuracy: 70.563%
Duration :57.048

Evaluate the model¶

In [33]:

# Create a Dataset object for the test set

test_dataset = tf.data.Dataset.from_tensor_slices((x_test, test_labels))
test_dataset = test_dataset.batch(32)

In [34]:

# Collect average loss and accuracy

epoch_loss_avg = tf.keras.metrics.Mean()
epoch_accuracy = tf.keras.metrics.CategoricalAccuracy()

In [35]:

# Loop over the test set and print scores

from tensorflow.keras.utils import to_categorical

for x, y in test_dataset:
    # Optimize the model
    loss_value = loss(model, x, y, weight_decay)    
    # Compute current loss
    epoch_loss_avg(loss_value)  
    # Compare predicted label to actual label
    epoch_accuracy(to_categorical(y), model(x))

print("Test loss: {:.3f}".format(epoch_loss_avg.result().numpy()))
print("Test accuracy: {:.3%}".format(epoch_accuracy.result().numpy()))

Test loss: 1.842
Test accuracy: 67.320%

Plot the learning curves¶

In [36]:

# Plot the training loss and accuracy

fig, axes = plt.subplots(2, sharex=True, figsize=(12, 8))
fig.suptitle('Training Metrics')

axes[0].set_ylabel("Loss", fontsize=14)
axes[0].plot(train_loss_results)

axes[1].set_ylabel("Accuracy", fontsize=14)
axes[1].set_xlabel("Epoch", fontsize=14)
axes[1].plot(train_accuracy_results)
plt.show()

Predict from the model¶

In [37]:

# Get the model prediction for an example input

predicted_label = np.argmax(model(x_train[np.newaxis,0]),axis=1)[0]
print("Prediction: {}".format(class_names[predicted_label]))
print("     Label: {}".format(class_names[train_labels[0]]))

Prediction: earn
     Label: earn

---

tf.function decorator¶

In [38]:

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Layer, Softmax
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.datasets import reuters
import numpy as np
import matplotlib.pyplot as plt
import time

Build the model¶

In [39]:

# Initialize a new model
model = MyModel(64, 64, 46)

Redefine the grad function using the @tf.function decorator¶

In [40]:

# Use the @tf.function decorator

@tf.function
def grad(model, inputs, targets, wd):
    with tf.GradientTape() as tape:
        loss_value = loss(model, inputs, targets, wd)
    return loss_value, tape.gradient(loss_value, model.trainable_variables)

Train the model¶

In [41]:

# Re-run the training loop

from tensorflow.keras.utils import to_categorical

start_time = time.time()

train_dataset = tf.data.Dataset.from_tensor_slices((x_train, train_labels))
train_dataset = train_dataset.batch(32)

# Keep results for plotting
train_loss_results = []
train_accuracy_results = []

num_epochs = 10
weight_decay = 0.005

for epoch in range(num_epochs):
    epoch_loss_avg = tf.keras.metrics.Mean()
    epoch_accuracy = tf.keras.metrics.CategoricalAccuracy()
    
    # Training loop
    for x, y in train_dataset:
        # Optimize the model
        loss_value, grads = grad(model, x, y, weight_decay)
        optimizer.apply_gradients(zip(grads, model.trainable_variables))
        
        # Compute current loop
        epoch_loss_avg(loss_value)
        # Compare predicted label to actual label
        epoch_accuracy(to_categorical(y), model(x))
        
    # End epoch
    train_loss_results.append(epoch_loss_avg.result())
    train_accuracy_results.append(epoch_accuracy.result())
    
    print('Epoch {:03d}: Loss {:.3f}, Accuracy: {:.3%}'.format(epoch, epoch_loss_avg.result(), epoch_accuracy.result()))
    
print("Duration :{:.3f}".format(time.time() - start_time))

WARNING:tensorflow:Layer my_model_3 is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.

If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

Epoch 000: Loss 2.441, Accuracy: 56.391%
Epoch 001: Loss 1.940, Accuracy: 64.885%
Epoch 002: Loss 1.841, Accuracy: 67.201%
Epoch 003: Loss 1.803, Accuracy: 68.214%
Epoch 004: Loss 1.765, Accuracy: 68.949%
Epoch 005: Loss 1.736, Accuracy: 69.016%
Epoch 006: Loss 1.726, Accuracy: 68.949%
Epoch 007: Loss 1.723, Accuracy: 69.238%
Epoch 008: Loss 1.709, Accuracy: 69.751%
Epoch 009: Loss 1.693, Accuracy: 69.851%
Duration :36.628

Print the autograph code¶

In [42]:

# Use tf.autograph.to_code to see the generated code
print(tf.autograph.to_code(grad.python_function))

def tf__grad(model, inputs, targets, wd):
    do_return = False
    retval_ = ag__.UndefinedReturnValue()
    with ag__.FunctionScope('grad', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:
        with tf.GradientTape() as tape:
            loss_value = ag__.converted_call(loss, (model, inputs, targets, wd), None, fscope)
        try:
            do_return = True
            retval_ = fscope.mark_return_value((loss_value, ag__.converted_call(tape.gradient, (loss_value, model.trainable_variables), None, fscope)))
        except:
            do_return = False
            raise
    (do_return,)
    return ag__.retval(retval_)